Jul 22 00:26:31.581055 ip-10-0-132-178 systemd[1]: Starting Kubernetes Kubelet... Jul 22 00:26:31.908834 ip-10-0-132-178 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 00:26:31.908834 ip-10-0-132-178 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 00:26:31.908834 ip-10-0-132-178 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 00:26:31.908834 ip-10-0-132-178 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 00:26:31.908834 ip-10-0-132-178 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 00:26:31.909430 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.908891 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 00:26:31.913500 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913484 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:31.913500 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913500 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913503 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913507 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913510 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913513 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913516 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913518 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913521 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913523 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913525 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913528 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913531 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913534 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913537 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913539 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913542 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913544 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913547 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913554 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913559 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:31.913566 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913563 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913565 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913568 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913571 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913574 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913577 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913580 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913583 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913585 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913588 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913590 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913593 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913595 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913598 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913601 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913606 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913609 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913612 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913614 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:31.914058 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913617 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913619 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913622 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913624 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913626 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913629 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913631 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913634 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913636 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913639 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913643 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913645 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913648 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913650 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913653 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913656 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913659 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913662 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913664 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913667 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:31.914538 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913669 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913672 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913674 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913677 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913679 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913682 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913684 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913687 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913690 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913692 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913695 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913697 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913700 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913703 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913705 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913708 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913710 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913713 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913715 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913718 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:31.915026 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913722 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913724 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913727 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913730 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913732 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913736 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913740 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913744 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913747 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913750 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913752 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913755 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913757 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913760 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913762 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913765 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913768 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913771 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.913773 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:31.915812 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914444 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914450 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914453 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914456 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914458 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914461 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914464 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914467 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914469 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914472 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914475 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914478 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914480 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914483 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914485 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914488 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914491 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914494 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914496 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914499 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914502 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:31.916350 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914505 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914507 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914510 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914513 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914515 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914518 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914521 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914523 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914525 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914528 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914530 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914533 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914535 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914537 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914540 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914543 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914545 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914548 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914550 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914553 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:31.916886 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914555 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914558 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914560 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914563 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914565 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914567 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914570 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914573 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914576 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914580 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914583 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914589 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914592 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914595 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914597 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914600 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914602 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914605 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914609 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:31.917364 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914613 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914616 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914619 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914621 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914625 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914629 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914631 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914635 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914638 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914640 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914644 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914647 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914649 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914652 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914654 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914657 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914659 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914662 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914665 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914667 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:31.917831 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914669 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914672 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914674 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914677 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914681 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914684 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914686 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914689 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914691 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914693 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914696 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914698 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914701 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914703 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914706 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914708 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914710 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914713 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.914715 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:31.918326 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915443 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915456 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915461 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915466 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915471 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915484 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915489 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915494 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915499 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915509 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915514 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915518 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915523 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915527 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915532 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915538 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915543 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915551 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915557 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915561 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:31.918838 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915565 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915569 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915578 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915583 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915587 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915592 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915596 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915601 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915604 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915608 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915615 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915621 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915626 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915631 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915641 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915646 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915651 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915656 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915660 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915665 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:31.919308 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915669 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915674 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915679 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915683 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915688 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915692 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915696 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915706 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915710 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915717 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915721 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915726 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915733 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915737 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915742 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915746 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915750 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915754 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915758 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:31.919796 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915768 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915772 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915795 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915800 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915805 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915810 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915817 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915823 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915828 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915833 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915840 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915849 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915853 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915858 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915862 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915867 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915871 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915876 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915880 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:31.920302 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915884 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915889 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915893 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915900 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915919 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915923 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915928 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915933 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915938 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915942 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915946 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915951 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915956 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915960 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915964 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915969 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915973 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915981 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915988 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915992 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:31.920797 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.915996 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916348 2567 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916366 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916386 2567 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916393 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916397 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916400 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916405 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916409 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916412 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916414 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916418 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916422 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916425 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916428 2567 flags.go:64] FLAG: --cgroup-root="" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916431 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916434 2567 flags.go:64] FLAG: --client-ca-file="" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916436 2567 flags.go:64] FLAG: --cloud-provider="external" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916439 2567 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916445 2567 flags.go:64] FLAG: --cluster-domain="" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916448 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916451 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916454 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916458 2567 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 00:26:31.921273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916461 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916464 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916467 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916470 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916473 2567 flags.go:64] FLAG: --contention-profiling="false" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916476 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916479 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916482 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916485 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916489 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916493 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916496 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916499 2567 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916502 2567 flags.go:64] FLAG: --enable-server="true" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916505 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916510 2567 flags.go:64] FLAG: --event-burst="100" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916513 2567 flags.go:64] FLAG: --event-qps="50" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916516 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916518 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916522 2567 flags.go:64] FLAG: --eviction-hard="" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916525 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916528 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916531 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916534 2567 flags.go:64] FLAG: --eviction-soft="" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916537 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 00:26:31.921977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916539 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916542 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916546 2567 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916548 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916551 2567 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916554 2567 flags.go:64] FLAG: --feature-gates="" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916558 2567 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916561 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916564 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916567 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916570 2567 flags.go:64] FLAG: --healthz-port="10248" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916573 2567 flags.go:64] FLAG: --help="false" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916575 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-132-178.ec2.internal" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916578 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916581 2567 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916584 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916588 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916594 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916598 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916601 2567 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916604 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916607 2567 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916610 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916613 2567 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 00:26:31.922578 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916616 2567 flags.go:64] FLAG: --kube-reserved="" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916618 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916621 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916624 2567 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916627 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916630 2567 flags.go:64] FLAG: --lock-file="" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916632 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916635 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916638 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916644 2567 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916647 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916650 2567 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916654 2567 flags.go:64] FLAG: --logging-format="text" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916657 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916660 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916662 2567 flags.go:64] FLAG: --manifest-url="" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916665 2567 flags.go:64] FLAG: --manifest-url-header="" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916669 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916672 2567 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916676 2567 flags.go:64] FLAG: --max-pods="110" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916679 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916681 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916684 2567 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916687 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916690 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 00:26:31.923539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916695 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916698 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916705 2567 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916708 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916711 2567 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916715 2567 flags.go:64] FLAG: --pod-cidr="" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916717 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916722 2567 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916725 2567 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916728 2567 flags.go:64] FLAG: --pods-per-core="0" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916731 2567 flags.go:64] FLAG: --port="10250" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916733 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916736 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-005efb6ce0cc8b2b6" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916739 2567 flags.go:64] FLAG: --qos-reserved="" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916742 2567 flags.go:64] FLAG: --read-only-port="10255" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916745 2567 flags.go:64] FLAG: --register-node="true" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916747 2567 flags.go:64] FLAG: --register-with-taints="" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916751 2567 flags.go:64] FLAG: --registry-burst="10" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916754 2567 flags.go:64] FLAG: --registry-qps="5" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916757 2567 flags.go:64] FLAG: --reserved-cpus="" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916760 2567 flags.go:64] FLAG: --reserved-memory="" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916764 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916767 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916770 2567 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916773 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 00:26:31.924633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916787 2567 flags.go:64] FLAG: --runonce="false" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916790 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916793 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916796 2567 flags.go:64] FLAG: --seccomp-default="false" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916799 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916802 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916805 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916809 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916812 2567 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916815 2567 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916818 2567 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916821 2567 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916824 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916827 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916830 2567 flags.go:64] FLAG: --system-cgroups="" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916832 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916838 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916841 2567 flags.go:64] FLAG: --tls-cert-file="" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916844 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916847 2567 flags.go:64] FLAG: --tls-min-version="" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916850 2567 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916853 2567 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916855 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916858 2567 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916861 2567 flags.go:64] FLAG: --v="2" Jul 22 00:26:31.925697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916865 2567 flags.go:64] FLAG: --version="false" Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916869 2567 flags.go:64] FLAG: --vmodule="" Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916874 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.916877 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.916990 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.916994 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.916997 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917001 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917004 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917006 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917009 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917011 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917015 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917017 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917021 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917024 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917026 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917029 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917032 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917035 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917038 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:31.926791 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917040 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917043 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917045 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917048 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917050 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917053 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917055 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917058 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917060 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917063 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917065 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917068 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917070 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917073 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917076 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917078 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917081 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917083 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917086 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917088 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:31.927452 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917091 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917093 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917095 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917098 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917102 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917106 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917109 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917112 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917116 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917118 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917121 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917124 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917126 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917128 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917131 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917134 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917136 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917139 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:31.928416 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917141 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917144 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917146 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917149 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917151 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917159 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917163 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917169 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917171 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917175 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917177 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917180 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917182 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917185 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917188 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917190 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917192 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917195 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917197 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917200 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:31.929033 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917203 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917205 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917208 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917210 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917213 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917215 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917218 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917220 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917223 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917226 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917228 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917231 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917234 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917236 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917239 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917241 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917244 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917246 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917249 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917252 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:31.929642 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917255 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:31.930181 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917258 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:31.930181 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917261 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:31.930181 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.917263 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:31.930181 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.918002 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 00:26:31.930181 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.925272 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 00:26:31.930181 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.925291 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 00:26:31.930181 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.925308 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 00:26:31.930181 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.925321 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 00:26:31.930181 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925406 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:31.930181 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925413 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:31.930181 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925419 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:31.930181 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925424 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:31.930181 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925429 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:31.930181 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925433 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:31.930181 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925438 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925442 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925447 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925451 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925455 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925459 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925463 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925467 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925472 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925477 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925481 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925485 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925489 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925494 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925498 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925502 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925506 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925510 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925515 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:31.930604 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925520 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925527 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925532 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925537 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925543 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925548 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925554 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925559 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925563 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925568 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925572 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925579 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925584 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925588 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925593 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925597 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925601 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925606 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925611 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:31.931115 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925615 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925619 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925624 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925628 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925632 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925636 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925640 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925644 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925648 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925653 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925657 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925661 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925666 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925670 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925675 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925679 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925683 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925690 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925694 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925698 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:31.931660 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925702 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925705 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925709 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925713 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925717 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925721 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925725 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925729 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925733 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925737 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925741 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925746 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925750 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925754 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925758 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925763 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925767 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925773 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925798 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925802 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:31.932258 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925806 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:31.932758 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925811 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:31.932758 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925815 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:31.932758 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925819 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:31.932758 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925823 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:31.932758 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925827 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:31.932758 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925832 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:31.932758 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925837 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:31.932758 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925842 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:31.932758 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925847 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:31.932758 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925852 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:31.932758 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925856 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:31.932758 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925861 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:31.932758 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925865 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:31.932758 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.925869 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:31.932758 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.925876 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926090 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926099 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926104 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926109 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926115 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926121 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926125 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926130 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926135 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926139 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926143 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926147 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926151 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926155 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926159 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926163 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926167 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926172 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:31.933170 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926176 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926180 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926184 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926189 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926193 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926197 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926203 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926207 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926212 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926216 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926221 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926225 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926229 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926233 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926237 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926241 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926246 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926250 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926254 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926258 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:31.933627 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926262 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926267 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926271 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926275 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926279 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926283 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926287 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926291 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926296 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926302 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926308 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926313 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926318 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926322 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926327 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926331 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926336 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926340 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926345 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926350 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:31.934139 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926355 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926359 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926363 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926367 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926370 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926377 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926381 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926386 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926390 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926394 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926398 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926402 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926407 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926411 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926415 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926419 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926423 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926448 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926455 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926461 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:31.934615 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926467 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926472 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926477 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926482 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926486 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926491 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926495 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926499 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926504 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926508 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926513 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926517 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926523 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926528 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926532 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926536 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926540 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926545 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926549 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:31.935126 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926553 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:31.935581 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:31.926557 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:31.935581 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.926565 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 00:26:31.935581 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.926703 2567 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 00:26:31.935581 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.929325 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 00:26:31.935581 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.930553 2567 server.go:1019] "Starting client certificate rotation" Jul 22 00:26:31.935581 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.930684 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 00:26:31.935581 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.931287 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 00:26:31.935581 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.932718 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 00:26:31.936647 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.936633 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 00:26:31.946359 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.946340 2567 log.go:25] "Validated CRI v1 runtime API" Jul 22 00:26:31.948927 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.948903 2567 log.go:25] "Validated CRI v1 image API" Jul 22 00:26:31.949715 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.949699 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 00:26:31.951650 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.951631 2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b1003ca8-f47d-4c33-909f-c3f15081c66d:/dev/nvme0n1p4 dbe147e7-b589-4160-9d94-a2d5a3f7a853:/dev/nvme0n1p3] Jul 22 00:26:31.951698 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.951650 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 00:26:31.957038 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.956934 2567 manager.go:216] Machine: {Timestamp:2026-07-22 00:26:31.955728409 +0000 UTC m=+0.281800603 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098922 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2537a3d695ea7f35a1c75fab0dfdda SystemUUID:ec2537a3-d695-ea7f-35a1-c75fab0dfdda BootID:473251df-0fb3-483a-9262-3bb1bb27cff4 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f6:b2:29:68:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f6:b2:29:68:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:62:c2:2f:01:84 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 00:26:31.957038 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.957033 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 00:26:31.957140 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.957128 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 00:26:31.959800 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.959761 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 00:26:31.959944 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.959804 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-178.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 00:26:31.960409 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.960399 2567 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 00:26:31.960442 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.960411 2567 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 00:26:31.960442 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.960418 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 00:26:31.960442 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.960432 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 00:26:31.961365 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.961346 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 00:26:31.961599 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.961588 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 00:26:31.965159 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.965148 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 22 00:26:31.965276 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.965266 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 00:26:31.965308 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.965296 2567 kubelet.go:486] "Attempting to sync node with API server" Jul 22 00:26:31.965336 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.965308 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 00:26:31.965336 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.965331 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 00:26:31.965395 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.965344 2567 kubelet.go:398] "Adding apiserver pod source" Jul 22 00:26:31.965395 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.965362 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 00:26:31.965755 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.965743 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 00:26:31.965816 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.965764 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 00:26:31.967587 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.967569 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 00:26:31.969517 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.969504 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 00:26:31.969575 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.969533 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 00:26:31.970605 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.970592 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 00:26:31.970660 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.970612 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 00:26:31.970660 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.970620 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 00:26:31.970660 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.970627 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 00:26:31.970660 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.970637 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 00:26:31.970660 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.970644 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 00:26:31.970660 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.970651 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 00:26:31.970660 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.970656 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 00:26:31.970660 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.970662 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 00:26:31.970918 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.970667 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 00:26:31.970918 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.970688 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 00:26:31.970918 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.970694 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 00:26:31.971884 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.971871 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 00:26:31.971927 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.971886 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 00:26:31.974725 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.974712 2567 server.go:1268] "Started kubelet" Jul 22 00:26:31.974807 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.974766 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 00:26:31.974884 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.974820 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 00:26:31.974937 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.974910 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 00:26:31.975543 ip-10-0-132-178 systemd[1]: Started Kubernetes Kubelet. Jul 22 00:26:31.976541 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:31.976516 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 00:26:31.976541 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.976535 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 00:26:31.976661 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:31.976587 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 00:26:31.976964 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.976945 2567 server.go:310] "Adding debug handlers to kubelet server" Jul 22 00:26:31.979813 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.979799 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 00:26:31.982117 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:31.981896 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 00:26:31.982234 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:31.981439 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-178.ec2.internal.18c474ff3f65f7a1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-178.ec2.internal,UID:ip-10-0-132-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-178.ec2.internal,},FirstTimestamp:2026-07-22 00:26:31.974688673 +0000 UTC m=+0.300760865,LastTimestamp:2026-07-22 00:26:31.974688673 +0000 UTC m=+0.300760865,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-178.ec2.internal,}" Jul 22 00:26:31.987411 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.987395 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 00:26:31.987499 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:31.987452 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-178.ec2.internal\" not found" Jul 22 00:26:31.987499 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.987477 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 00:26:31.987963 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.987934 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 00:26:31.987963 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.987936 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 00:26:31.987963 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.987962 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 00:26:31.988129 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.988046 2567 reconstruct.go:97] "Volume reconstruction finished" Jul 22 00:26:31.988129 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.988054 2567 reconciler.go:29] "Reconciler: start to sync state" Jul 22 00:26:31.988129 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.988096 2567 factory.go:55] Registering systemd factory Jul 22 00:26:31.988129 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.988110 2567 factory.go:223] Registration of the systemd container factory successfully Jul 22 00:26:31.988301 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.988288 2567 factory.go:153] Registering CRI-O factory Jul 22 00:26:31.988353 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.988304 2567 factory.go:223] Registration of the crio container factory successfully Jul 22 00:26:31.988396 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.988352 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 00:26:31.988396 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.988378 2567 factory.go:103] Registering Raw factory Jul 22 00:26:31.988396 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.988391 2567 manager.go:1196] Started watching for new ooms in manager Jul 22 00:26:31.988699 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.988687 2567 manager.go:349] Starting recovery of all containers Jul 22 00:26:31.989336 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:31.989287 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 00:26:31.989516 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:31.989495 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 00:26:31.998676 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:31.998660 2567 manager.go:354] Recovery completed Jul 22 00:26:32.002701 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.002690 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:32.004943 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.004922 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:32.005024 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.004954 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:32.005024 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.004964 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:32.005473 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.005453 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 00:26:32.005473 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.005469 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:32.005597 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.005471 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 00:26:32.005597 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.005540 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 22 00:26:32.006196 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.006180 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:32.006257 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.006210 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:32.006257 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.006225 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:32.006709 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.006693 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nqdtt" Jul 22 00:26:32.007463 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.007402 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-178.ec2.internal.18c474ff413395b2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-178.ec2.internal,UID:ip-10-0-132-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-178.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-178.ec2.internal,},FirstTimestamp:2026-07-22 00:26:32.004941234 +0000 UTC m=+0.331013427,LastTimestamp:2026-07-22 00:26:32.004941234 +0000 UTC m=+0.331013427,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-178.ec2.internal,}" Jul 22 00:26:32.007923 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.007899 2567 policy_none.go:49] "None policy: Start" Jul 22 00:26:32.007923 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.007923 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 00:26:32.008037 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.007933 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 00:26:32.009848 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.009835 2567 policy_none.go:47] "Start" Jul 22 00:26:32.013971 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.013949 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nqdtt" Jul 22 00:26:32.019019 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.018959 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-178.ec2.internal.18c474ff4133d98d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-178.ec2.internal,UID:ip-10-0-132-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-178.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-178.ec2.internal,},FirstTimestamp:2026-07-22 00:26:32.004958605 +0000 UTC m=+0.331030798,LastTimestamp:2026-07-22 00:26:32.004958605 +0000 UTC m=+0.331030798,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-178.ec2.internal,}" Jul 22 00:26:32.046369 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.046349 2567 manager.go:337] "Starting Device Plugin manager" Jul 22 00:26:32.064720 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.046385 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 00:26:32.064720 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.046395 2567 server.go:87] "Starting device plugin registration server" Jul 22 00:26:32.064720 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.046670 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 00:26:32.064720 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.046696 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 00:26:32.064720 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.046792 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 00:26:32.064720 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.046874 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 00:26:32.064720 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.046882 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 00:26:32.064720 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.047355 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 00:26:32.064720 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.047402 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-178.ec2.internal\" not found" Jul 22 00:26:32.110581 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.110462 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 00:26:32.111740 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.111722 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 00:26:32.111817 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.111768 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 00:26:32.111868 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.111831 2567 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 00:26:32.111921 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.111872 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 00:26:32.113972 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.113951 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 00:26:32.147672 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.147655 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:32.148400 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.148385 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:32.148496 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.148409 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:32.148496 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.148419 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:32.148496 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.148444 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.163492 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.163478 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.163560 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.163498 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-178.ec2.internal\": node \"ip-10-0-132-178.ec2.internal\" not found" Jul 22 00:26:32.181723 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.181701 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-178.ec2.internal\" not found" Jul 22 00:26:32.212216 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.212187 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-178.ec2.internal"] Jul 22 00:26:32.212291 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.212249 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:32.213498 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.213477 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:32.213582 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.213506 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:32.213582 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.213516 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:32.215916 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.215897 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:32.216051 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.216038 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.216085 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.216077 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:32.216576 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.216563 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:32.216640 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.216584 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:32.216640 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.216606 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:32.216640 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.216615 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:32.216756 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.216590 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:32.216756 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.216668 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:32.218947 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.218931 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.219020 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.218955 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:32.219558 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.219544 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:32.219630 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.219572 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:32.219630 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.219582 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:32.233308 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.233289 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-178.ec2.internal\" not found" node="ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.237266 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.237252 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-178.ec2.internal\" not found" node="ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.282343 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.282301 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-178.ec2.internal\" not found" Jul 22 00:26:32.289717 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.289699 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cbc03ea7b9c0120a2d5366cd2de1b0d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal\" (UID: \"cbc03ea7b9c0120a2d5366cd2de1b0d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.289765 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.289722 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbc03ea7b9c0120a2d5366cd2de1b0d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal\" (UID: \"cbc03ea7b9c0120a2d5366cd2de1b0d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.289765 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.289738 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c36159d4fd5288301a0516429f295afe-config\") pod \"kube-apiserver-proxy-ip-10-0-132-178.ec2.internal\" (UID: \"c36159d4fd5288301a0516429f295afe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.383071 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.383039 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-178.ec2.internal\" not found" Jul 22 00:26:32.390395 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.390378 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cbc03ea7b9c0120a2d5366cd2de1b0d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal\" (UID: \"cbc03ea7b9c0120a2d5366cd2de1b0d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.390446 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.390401 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbc03ea7b9c0120a2d5366cd2de1b0d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal\" (UID: \"cbc03ea7b9c0120a2d5366cd2de1b0d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.390446 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.390418 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c36159d4fd5288301a0516429f295afe-config\") pod \"kube-apiserver-proxy-ip-10-0-132-178.ec2.internal\" (UID: \"c36159d4fd5288301a0516429f295afe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.390507 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.390458 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c36159d4fd5288301a0516429f295afe-config\") pod \"kube-apiserver-proxy-ip-10-0-132-178.ec2.internal\" (UID: \"c36159d4fd5288301a0516429f295afe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.390507 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.390471 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cbc03ea7b9c0120a2d5366cd2de1b0d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal\" (UID: \"cbc03ea7b9c0120a2d5366cd2de1b0d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.390507 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.390486 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbc03ea7b9c0120a2d5366cd2de1b0d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal\" (UID: \"cbc03ea7b9c0120a2d5366cd2de1b0d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.483795 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.483744 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-178.ec2.internal\" not found" Jul 22 00:26:32.536299 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.536255 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.539579 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.539559 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-178.ec2.internal" Jul 22 00:26:32.584452 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.584421 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-178.ec2.internal\" not found" Jul 22 00:26:32.684953 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.684929 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-178.ec2.internal\" not found" Jul 22 00:26:32.785401 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.785378 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-178.ec2.internal\" not found" Jul 22 00:26:32.885904 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.885850 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-178.ec2.internal\" not found" Jul 22 00:26:32.930294 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.930276 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 00:26:32.930643 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.930396 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 00:26:32.986815 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:32.986773 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-178.ec2.internal\" not found" Jul 22 00:26:32.992558 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:32.992543 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 00:26:33.019067 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.019039 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 00:21:32 +0000 UTC" deadline="2028-02-10 16:37:52.164921795 +0000 UTC" Jul 22 00:26:33.019067 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.019063 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13648h11m19.145861595s" Jul 22 00:26:33.035492 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.035464 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 00:26:33.044632 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.044615 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 00:26:33.087621 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.087596 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal" Jul 22 00:26:33.099133 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.099119 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 00:26:33.099898 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.099884 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-178.ec2.internal" Jul 22 00:26:33.108008 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.107990 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 00:26:33.184632 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:33.184597 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36159d4fd5288301a0516429f295afe.slice/crio-02b6dec3fb707e959771e298431aebfa3ec7da5fff2f20386bd10ae02d89b70a WatchSource:0}: Error finding container 02b6dec3fb707e959771e298431aebfa3ec7da5fff2f20386bd10ae02d89b70a: Status 404 returned error can't find the container with id 02b6dec3fb707e959771e298431aebfa3ec7da5fff2f20386bd10ae02d89b70a Jul 22 00:26:33.189206 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.189186 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 00:26:33.461915 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:33.461886 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc03ea7b9c0120a2d5366cd2de1b0d7.slice/crio-243ebc7085ff2564fe85bf92f19e9d91eb0182a9d79e7f847ae3e7d25e7cd042 WatchSource:0}: Error finding container 243ebc7085ff2564fe85bf92f19e9d91eb0182a9d79e7f847ae3e7d25e7cd042: Status 404 returned error can't find the container with id 243ebc7085ff2564fe85bf92f19e9d91eb0182a9d79e7f847ae3e7d25e7cd042 Jul 22 00:26:33.966403 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.966375 2567 apiserver.go:52] "Watching apiserver" Jul 22 00:26:33.975421 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.975396 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 00:26:33.976669 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.976635 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-qtzz2","openshift-network-operator/iptables-alerter-pxvv8","kube-system/global-pull-secret-syncer-trczs","openshift-cluster-node-tuning-operator/tuned-zzww7","openshift-dns/node-resolver-kxz6k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal","openshift-multus/multus-additional-cni-plugins-5pqtv","openshift-multus/network-metrics-daemon-6q2tc","openshift-ovn-kubernetes/ovnkube-node-g5jvn","kube-system/konnectivity-agent-4wfbw","kube-system/kube-apiserver-proxy-ip-10-0-132-178.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw","openshift-image-registry/node-ca-lv725","openshift-multus/multus-gvw2z"] Jul 22 00:26:33.979446 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.979422 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:33.981811 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.981769 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lv725" Jul 22 00:26:33.982422 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.982401 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 00:26:33.982518 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.982451 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 00:26:33.982518 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.982462 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 00:26:33.982518 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.982476 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-thkc2\"" Jul 22 00:26:33.982673 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.982400 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 00:26:33.982673 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.982614 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 00:26:33.984146 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.984118 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:33.984248 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.984200 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 00:26:33.984248 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:33.984198 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:26:33.984359 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.984312 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 00:26:33.984479 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.984461 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sprbv\"" Jul 22 00:26:33.984732 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.984706 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 00:26:33.986636 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.986612 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:33.988010 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.987981 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 00:26:33.989694 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.989080 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 00:26:33.989694 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.989140 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 00:26:33.989694 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.989337 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kxz6k" Jul 22 00:26:33.989694 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.989433 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z64sk\"" Jul 22 00:26:33.991402 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.991383 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gvw2z" Jul 22 00:26:33.991849 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.991729 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 00:26:33.991969 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.991899 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 00:26:33.992389 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.992228 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-spqlp\"" Jul 22 00:26:33.993776 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.993757 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 00:26:33.994145 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.994120 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:33.994284 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.994267 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dvpfm\"" Jul 22 00:26:33.996936 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.996919 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 00:26:33.997263 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.997245 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:33.997355 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:33.997319 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:26:33.997558 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.997539 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 00:26:33.997639 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.997612 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 00:26:33.997731 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.997711 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wxkc6\"" Jul 22 00:26:33.997798 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.997774 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-dbus\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:33.997875 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.997803 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-var-lib-kubelet\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:33.997939 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.997900 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5-hosts-file\") pod \"node-resolver-kxz6k\" (UID: \"c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5\") " pod="openshift-dns/node-resolver-kxz6k" Jul 22 00:26:33.997993 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.997928 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5-tmp-dir\") pod \"node-resolver-kxz6k\" (UID: \"c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5\") " pod="openshift-dns/node-resolver-kxz6k" Jul 22 00:26:33.997993 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.997960 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 00:26:33.997993 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.997965 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66d2c403-f38d-4c15-b761-8d50c9f3b142-cnibin\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:33.997993 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.997988 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-tuned\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:33.998200 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998012 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7pvj\" (UniqueName: \"kubernetes.io/projected/c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5-kube-api-access-x7pvj\") pod \"node-resolver-kxz6k\" (UID: \"c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5\") " pod="openshift-dns/node-resolver-kxz6k" Jul 22 00:26:33.998200 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998037 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 00:26:33.998200 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998051 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66d2c403-f38d-4c15-b761-8d50c9f3b142-cni-binary-copy\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:33.998200 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998123 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 00:26:33.998200 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998146 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66d2c403-f38d-4c15-b761-8d50c9f3b142-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:33.998200 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998177 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-sysconfig\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:33.998500 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998204 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-sysctl-conf\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:33.998500 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998227 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2w5\" (UniqueName: \"kubernetes.io/projected/5dd30122-7106-4258-af8e-b3c91bf3debe-kube-api-access-zr2w5\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:33.998733 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998253 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66d2c403-f38d-4c15-b761-8d50c9f3b142-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:33.998814 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998752 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d60d84ba-a0ae-4e46-9fdf-7830c7d5e798-serviceca\") pod \"node-ca-lv725\" (UID: \"d60d84ba-a0ae-4e46-9fdf-7830c7d5e798\") " pod="openshift-image-registry/node-ca-lv725" Jul 22 00:26:33.998814 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998778 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:33.998814 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998804 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-modprobe-d\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:33.999004 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998831 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66d2c403-f38d-4c15-b761-8d50c9f3b142-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:33.999004 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998865 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d60d84ba-a0ae-4e46-9fdf-7830c7d5e798-host\") pod \"node-ca-lv725\" (UID: \"d60d84ba-a0ae-4e46-9fdf-7830c7d5e798\") " pod="openshift-image-registry/node-ca-lv725" Jul 22 00:26:33.999004 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998896 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-sysctl-d\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:33.999004 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998931 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/66d2c403-f38d-4c15-b761-8d50c9f3b142-os-release\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:33.999004 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998972 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-systemd\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:33.999004 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.998999 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-run\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:33.999299 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.999024 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-sys\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:33.999299 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.999051 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvblq\" (UniqueName: \"kubernetes.io/projected/66d2c403-f38d-4c15-b761-8d50c9f3b142-kube-api-access-pvblq\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:33.999299 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.999078 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzczz\" (UniqueName: \"kubernetes.io/projected/d60d84ba-a0ae-4e46-9fdf-7830c7d5e798-kube-api-access-qzczz\") pod \"node-ca-lv725\" (UID: \"d60d84ba-a0ae-4e46-9fdf-7830c7d5e798\") " pod="openshift-image-registry/node-ca-lv725" Jul 22 00:26:33.999299 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.999103 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-kubelet-config\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:33.999299 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.999122 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-host\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:33.999299 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.999135 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dd30122-7106-4258-af8e-b3c91bf3debe-tmp\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:33.999299 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.999152 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66d2c403-f38d-4c15-b761-8d50c9f3b142-system-cni-dir\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:33.999299 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.999175 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-kubernetes\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:33.999299 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.999223 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-lib-modules\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:33.999872 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:33.999833 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pxvv8" Jul 22 00:26:34.001386 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.001361 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 00:26:34.002318 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.002299 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 00:26:34.002483 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.002430 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hczsn\"" Jul 22 00:26:34.002951 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.002602 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 00:26:34.002951 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.002690 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4wfbw" Jul 22 00:26:34.002951 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.002947 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 00:26:34.005788 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.005086 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 00:26:34.005788 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.005306 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 00:26:34.005788 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.005441 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:34.005788 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.005308 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9wdn6\"" Jul 22 00:26:34.005788 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.005513 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:26:34.008927 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.007917 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.011142 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.010697 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 00:26:34.011142 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.010746 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 00:26:34.011142 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.010702 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9nf5t\"" Jul 22 00:26:34.011142 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.011069 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 00:26:34.027187 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.027169 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zmn6j" Jul 22 00:26:34.037223 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.037189 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zmn6j" Jul 22 00:26:34.089564 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.089544 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 00:26:34.100085 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100065 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5-tmp-dir\") pod \"node-resolver-kxz6k\" (UID: \"c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5\") " pod="openshift-dns/node-resolver-kxz6k" Jul 22 00:26:34.100206 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100098 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45l7k\" (UniqueName: \"kubernetes.io/projected/83fc1f05-76b5-4574-82ad-e12760548aef-kube-api-access-45l7k\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.100206 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100128 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-run-netns\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.100206 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100152 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-cni-bin\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.100206 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100171 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-tuned\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.100206 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100193 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7pvj\" (UniqueName: \"kubernetes.io/projected/c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5-kube-api-access-x7pvj\") pod \"node-resolver-kxz6k\" (UID: \"c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5\") " pod="openshift-dns/node-resolver-kxz6k" Jul 22 00:26:34.100466 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100218 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-etc-openvswitch\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.100466 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100239 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-run-openvswitch\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.100466 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100262 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-multus-cni-dir\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.100466 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100285 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-multus-conf-dir\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.100466 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100316 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66d2c403-f38d-4c15-b761-8d50c9f3b142-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.100466 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100347 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-device-dir\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.100466 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100388 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-var-lib-openvswitch\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.100466 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100414 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66d2c403-f38d-4c15-b761-8d50c9f3b142-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.100466 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100430 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5-tmp-dir\") pod \"node-resolver-kxz6k\" (UID: \"c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5\") " pod="openshift-dns/node-resolver-kxz6k" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100438 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d60d84ba-a0ae-4e46-9fdf-7830c7d5e798-serviceca\") pod \"node-ca-lv725\" (UID: \"d60d84ba-a0ae-4e46-9fdf-7830c7d5e798\") " pod="openshift-image-registry/node-ca-lv725" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100502 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100661 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-modprobe-d\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100685 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66d2c403-f38d-4c15-b761-8d50c9f3b142-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100696 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-registration-dir\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100726 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-multus-daemon-config\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100754 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d60d84ba-a0ae-4e46-9fdf-7830c7d5e798-host\") pod \"node-ca-lv725\" (UID: \"d60d84ba-a0ae-4e46-9fdf-7830c7d5e798\") " pod="openshift-image-registry/node-ca-lv725" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100769 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-modprobe-d\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100777 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-cnibin\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100842 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d60d84ba-a0ae-4e46-9fdf-7830c7d5e798-host\") pod \"node-ca-lv725\" (UID: \"d60d84ba-a0ae-4e46-9fdf-7830c7d5e798\") " pod="openshift-image-registry/node-ca-lv725" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100871 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-hostroot\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100902 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhtgk\" (UniqueName: \"kubernetes.io/projected/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-kube-api-access-rhtgk\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100930 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07ca19d0-5de0-4391-b15e-2a229d853303-ovnkube-config\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100961 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9678f689-061e-4efc-94b2-890933759edd-iptables-alerter-script\") pod \"iptables-alerter-pxvv8\" (UID: \"9678f689-061e-4efc-94b2-890933759edd\") " pod="openshift-network-operator/iptables-alerter-pxvv8" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100956 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d60d84ba-a0ae-4e46-9fdf-7830c7d5e798-serviceca\") pod \"node-ca-lv725\" (UID: \"d60d84ba-a0ae-4e46-9fdf-7830c7d5e798\") " pod="openshift-image-registry/node-ca-lv725" Jul 22 00:26:34.100983 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.100991 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f1ef826a-11c1-4baf-82fc-134c519b472a-konnectivity-ca\") pod \"konnectivity-agent-4wfbw\" (UID: \"f1ef826a-11c1-4baf-82fc-134c519b472a\") " pod="kube-system/konnectivity-agent-4wfbw" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101021 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-systemd\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101047 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-sys\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101070 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-slash\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101077 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-systemd\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101095 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-log-socket\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101133 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07ca19d0-5de0-4391-b15e-2a229d853303-env-overrides\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101147 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-sys\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101162 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9678f689-061e-4efc-94b2-890933759edd-host-slash\") pod \"iptables-alerter-pxvv8\" (UID: \"9678f689-061e-4efc-94b2-890933759edd\") " pod="openshift-network-operator/iptables-alerter-pxvv8" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101166 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66d2c403-f38d-4c15-b761-8d50c9f3b142-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101193 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f1ef826a-11c1-4baf-82fc-134c519b472a-agent-certs\") pod \"konnectivity-agent-4wfbw\" (UID: \"f1ef826a-11c1-4baf-82fc-134c519b472a\") " pod="kube-system/konnectivity-agent-4wfbw" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101227 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvblq\" (UniqueName: \"kubernetes.io/projected/66d2c403-f38d-4c15-b761-8d50c9f3b142-kube-api-access-pvblq\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101299 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-kubelet-config\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101367 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-kubelet-config\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101376 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-host\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101406 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-run-ovn-kubernetes\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101419 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-host\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.101882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-cni-netd\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101465 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66d2c403-f38d-4c15-b761-8d50c9f3b142-system-cni-dir\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101525 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66d2c403-f38d-4c15-b761-8d50c9f3b142-system-cni-dir\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101534 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-lib-modules\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101565 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-var-lib-cni-bin\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101593 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07ca19d0-5de0-4391-b15e-2a229d853303-ovn-node-metrics-cert\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101619 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-dbus\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101640 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-var-lib-kubelet\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101660 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5-hosts-file\") pod \"node-resolver-kxz6k\" (UID: \"c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5\") " pod="openshift-dns/node-resolver-kxz6k" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101680 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-run-systemd\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101697 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-lib-modules\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101702 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66d2c403-f38d-4c15-b761-8d50c9f3b142-cnibin\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101743 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-var-lib-cni-multus\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101765 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-run-multus-certs\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101782 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-etc-kubernetes\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101793 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5-hosts-file\") pod \"node-resolver-kxz6k\" (UID: \"c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5\") " pod="openshift-dns/node-resolver-kxz6k" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101747 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66d2c403-f38d-4c15-b761-8d50c9f3b142-cnibin\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.102659 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101798 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-run-ovn\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101829 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-dbus\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101858 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-var-lib-kubelet\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101901 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07ca19d0-5de0-4391-b15e-2a229d853303-ovnkube-script-lib\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101923 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-sys-fs\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101953 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66d2c403-f38d-4c15-b761-8d50c9f3b142-cni-binary-copy\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.101993 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-sysconfig\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102042 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-sysctl-conf\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102053 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-sysconfig\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102071 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2w5\" (UniqueName: \"kubernetes.io/projected/5dd30122-7106-4258-af8e-b3c91bf3debe-kube-api-access-zr2w5\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102095 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102144 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-etc-selinux\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102166 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-run-netns\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102175 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-sysctl-conf\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102188 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102230 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66d2c403-f38d-4c15-b761-8d50c9f3b142-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102253 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-sysctl-d\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.103528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102274 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-multus-socket-dir-parent\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102296 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-var-lib-kubelet\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102317 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-kubelet\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102340 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102375 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/66d2c403-f38d-4c15-b761-8d50c9f3b142-os-release\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102397 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-run\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102417 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-socket-dir\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102427 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66d2c403-f38d-4c15-b761-8d50c9f3b142-cni-binary-copy\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102476 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-os-release\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102506 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-sysctl-d\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102508 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-run-k8s-cni-cncf-io\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102540 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/66d2c403-f38d-4c15-b761-8d50c9f3b142-os-release\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102558 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-run\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102602 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-systemd-units\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102638 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wsfp\" (UniqueName: \"kubernetes.io/projected/9678f689-061e-4efc-94b2-890933759edd-kube-api-access-6wsfp\") pod \"iptables-alerter-pxvv8\" (UID: \"9678f689-061e-4efc-94b2-890933759edd\") " pod="openshift-network-operator/iptables-alerter-pxvv8" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.102693 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102709 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzczz\" (UniqueName: \"kubernetes.io/projected/d60d84ba-a0ae-4e46-9fdf-7830c7d5e798-kube-api-access-qzczz\") pod \"node-ca-lv725\" (UID: \"d60d84ba-a0ae-4e46-9fdf-7830c7d5e798\") " pod="openshift-image-registry/node-ca-lv725" Jul 22 00:26:34.104412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102738 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dd30122-7106-4258-af8e-b3c91bf3debe-tmp\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.105291 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.102799 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret podName:30d68a8f-032a-4d0e-90bf-6f70e5b548f2 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:34.602753895 +0000 UTC m=+2.928826097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret") pod "global-pull-secret-syncer-trczs" (UID: "30d68a8f-032a-4d0e-90bf-6f70e5b548f2") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:34.105291 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102822 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x68d\" (UniqueName: \"kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d\") pod \"network-check-target-qtzz2\" (UID: \"7c8c37c8-072f-4c0b-84a5-e880dcbb7d17\") " pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:34.105291 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102884 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-cni-binary-copy\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.105291 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102908 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66d2c403-f38d-4c15-b761-8d50c9f3b142-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.105291 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102957 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-node-log\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.105291 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.102983 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:34.105291 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.103011 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-kubernetes\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.105291 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.103091 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-system-cni-dir\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.105291 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.103123 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-kubernetes\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.105291 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.103131 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcclf\" (UniqueName: \"kubernetes.io/projected/07ca19d0-5de0-4391-b15e-2a229d853303-kube-api-access-fcclf\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.105291 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.103159 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsm87\" (UniqueName: \"kubernetes.io/projected/d55898f6-dc0e-408a-99cc-9883ae591ac5-kube-api-access-qsm87\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:34.105291 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.104851 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5dd30122-7106-4258-af8e-b3c91bf3debe-etc-tuned\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.105291 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.105082 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dd30122-7106-4258-af8e-b3c91bf3debe-tmp\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.108188 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.108168 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7pvj\" (UniqueName: \"kubernetes.io/projected/c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5-kube-api-access-x7pvj\") pod \"node-resolver-kxz6k\" (UID: \"c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5\") " pod="openshift-dns/node-resolver-kxz6k" Jul 22 00:26:34.109819 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.109791 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvblq\" (UniqueName: \"kubernetes.io/projected/66d2c403-f38d-4c15-b761-8d50c9f3b142-kube-api-access-pvblq\") pod \"multus-additional-cni-plugins-5pqtv\" (UID: \"66d2c403-f38d-4c15-b761-8d50c9f3b142\") " pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.116830 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.116808 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2w5\" (UniqueName: \"kubernetes.io/projected/5dd30122-7106-4258-af8e-b3c91bf3debe-kube-api-access-zr2w5\") pod \"tuned-zzww7\" (UID: \"5dd30122-7106-4258-af8e-b3c91bf3debe\") " pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.116937 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.116879 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzczz\" (UniqueName: \"kubernetes.io/projected/d60d84ba-a0ae-4e46-9fdf-7830c7d5e798-kube-api-access-qzczz\") pod \"node-ca-lv725\" (UID: \"d60d84ba-a0ae-4e46-9fdf-7830c7d5e798\") " pod="openshift-image-registry/node-ca-lv725" Jul 22 00:26:34.117167 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.117102 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal" event={"ID":"cbc03ea7b9c0120a2d5366cd2de1b0d7","Type":"ContainerStarted","Data":"243ebc7085ff2564fe85bf92f19e9d91eb0182a9d79e7f847ae3e7d25e7cd042"} Jul 22 00:26:34.118407 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.118384 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-178.ec2.internal" event={"ID":"c36159d4fd5288301a0516429f295afe","Type":"ContainerStarted","Data":"02b6dec3fb707e959771e298431aebfa3ec7da5fff2f20386bd10ae02d89b70a"} Jul 22 00:26:34.203915 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.203880 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-system-cni-dir\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.204058 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.203919 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcclf\" (UniqueName: \"kubernetes.io/projected/07ca19d0-5de0-4391-b15e-2a229d853303-kube-api-access-fcclf\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.204058 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.203972 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsm87\" (UniqueName: \"kubernetes.io/projected/d55898f6-dc0e-408a-99cc-9883ae591ac5-kube-api-access-qsm87\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:34.204058 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204005 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45l7k\" (UniqueName: \"kubernetes.io/projected/83fc1f05-76b5-4574-82ad-e12760548aef-kube-api-access-45l7k\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.204058 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204023 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-system-cni-dir\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.204058 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204028 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-run-netns\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.204337 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204072 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-run-netns\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.204337 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204081 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-cni-bin\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.204337 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204117 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-etc-openvswitch\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.204337 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204122 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-cni-bin\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.204337 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204165 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-etc-openvswitch\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.204337 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204195 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-run-openvswitch\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.204337 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204237 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-multus-cni-dir\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.204337 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204240 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-run-openvswitch\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.204337 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204259 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-multus-conf-dir\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.204337 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204277 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-device-dir\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.204337 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204292 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-multus-conf-dir\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.204337 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204298 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-var-lib-openvswitch\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.204337 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204309 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-multus-cni-dir\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.204337 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204323 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-registration-dir\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204343 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-device-dir\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204364 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-multus-daemon-config\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204390 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-var-lib-openvswitch\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204394 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-cnibin\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204390 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-registration-dir\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204435 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-hostroot\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204447 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-cnibin\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204461 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhtgk\" (UniqueName: \"kubernetes.io/projected/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-kube-api-access-rhtgk\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204488 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07ca19d0-5de0-4391-b15e-2a229d853303-ovnkube-config\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204513 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9678f689-061e-4efc-94b2-890933759edd-iptables-alerter-script\") pod \"iptables-alerter-pxvv8\" (UID: \"9678f689-061e-4efc-94b2-890933759edd\") " pod="openshift-network-operator/iptables-alerter-pxvv8" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204488 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-hostroot\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204537 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f1ef826a-11c1-4baf-82fc-134c519b472a-konnectivity-ca\") pod \"konnectivity-agent-4wfbw\" (UID: \"f1ef826a-11c1-4baf-82fc-134c519b472a\") " pod="kube-system/konnectivity-agent-4wfbw" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204579 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-slash\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204602 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-log-socket\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204626 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07ca19d0-5de0-4391-b15e-2a229d853303-env-overrides\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204659 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9678f689-061e-4efc-94b2-890933759edd-host-slash\") pod \"iptables-alerter-pxvv8\" (UID: \"9678f689-061e-4efc-94b2-890933759edd\") " pod="openshift-network-operator/iptables-alerter-pxvv8" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204690 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-slash\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.205092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204702 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f1ef826a-11c1-4baf-82fc-134c519b472a-agent-certs\") pod \"konnectivity-agent-4wfbw\" (UID: \"f1ef826a-11c1-4baf-82fc-134c519b472a\") " pod="kube-system/konnectivity-agent-4wfbw" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204731 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-run-ovn-kubernetes\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204755 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-cni-netd\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204780 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-var-lib-cni-bin\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204803 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07ca19d0-5de0-4391-b15e-2a229d853303-ovn-node-metrics-cert\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204847 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-run-systemd\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204887 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-var-lib-cni-multus\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204912 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-run-multus-certs\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204937 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-etc-kubernetes\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204961 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-run-ovn\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.204985 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07ca19d0-5de0-4391-b15e-2a229d853303-ovnkube-script-lib\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205039 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-multus-daemon-config\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205060 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-var-lib-cni-bin\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205134 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-var-lib-cni-multus\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205143 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-sys-fs\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205184 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205192 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-sys-fs\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.206032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205216 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-etc-selinux\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205221 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9678f689-061e-4efc-94b2-890933759edd-iptables-alerter-script\") pod \"iptables-alerter-pxvv8\" (UID: \"9678f689-061e-4efc-94b2-890933759edd\") " pod="openshift-network-operator/iptables-alerter-pxvv8" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205244 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-run-netns\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205264 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-log-socket\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205294 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-multus-socket-dir-parent\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205304 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-run-multus-certs\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205327 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-var-lib-kubelet\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205343 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f1ef826a-11c1-4baf-82fc-134c519b472a-konnectivity-ca\") pod \"konnectivity-agent-4wfbw\" (UID: \"f1ef826a-11c1-4baf-82fc-134c519b472a\") " pod="kube-system/konnectivity-agent-4wfbw" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205352 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-kubelet\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205361 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-run-ovn\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205379 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205407 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-run-netns\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205411 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-socket-dir\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205437 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-os-release\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205463 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-run-k8s-cni-cncf-io\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205490 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-systemd-units\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205516 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wsfp\" (UniqueName: \"kubernetes.io/projected/9678f689-061e-4efc-94b2-890933759edd-kube-api-access-6wsfp\") pod \"iptables-alerter-pxvv8\" (UID: \"9678f689-061e-4efc-94b2-890933759edd\") " pod="openshift-network-operator/iptables-alerter-pxvv8" Jul 22 00:26:34.206904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205544 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x68d\" (UniqueName: \"kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d\") pod \"network-check-target-qtzz2\" (UID: \"7c8c37c8-072f-4c0b-84a5-e880dcbb7d17\") " pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205574 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-cni-binary-copy\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205603 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-node-log\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205623 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07ca19d0-5de0-4391-b15e-2a229d853303-ovnkube-script-lib\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205631 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.205719 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205725 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9678f689-061e-4efc-94b2-890933759edd-host-slash\") pod \"iptables-alerter-pxvv8\" (UID: \"9678f689-061e-4efc-94b2-890933759edd\") " pod="openshift-network-operator/iptables-alerter-pxvv8" Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205729 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205767 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07ca19d0-5de0-4391-b15e-2a229d853303-env-overrides\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205787 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-run-ovn-kubernetes\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205797 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-run-systemd\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.205776 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs podName:d55898f6-dc0e-408a-99cc-9883ae591ac5 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:34.705759548 +0000 UTC m=+3.031831735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs") pod "network-metrics-daemon-6q2tc" (UID: "d55898f6-dc0e-408a-99cc-9883ae591ac5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205853 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-cni-netd\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205873 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-etc-kubernetes\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205905 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-kubelet\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205943 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-multus-socket-dir-parent\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205960 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-var-lib-kubelet\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.207780 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.205969 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-etc-selinux\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.208604 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.206014 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-node-log\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.208604 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.206012 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.208604 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.206047 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-host-run-k8s-cni-cncf-io\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.208604 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.206081 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/83fc1f05-76b5-4574-82ad-e12760548aef-socket-dir\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.208604 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.206109 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-os-release\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.208604 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.206148 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07ca19d0-5de0-4391-b15e-2a229d853303-systemd-units\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.208604 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.206431 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-cni-binary-copy\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.208604 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.206472 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07ca19d0-5de0-4391-b15e-2a229d853303-ovnkube-config\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.208604 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.208242 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07ca19d0-5de0-4391-b15e-2a229d853303-ovn-node-metrics-cert\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.208604 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.208524 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f1ef826a-11c1-4baf-82fc-134c519b472a-agent-certs\") pod \"konnectivity-agent-4wfbw\" (UID: \"f1ef826a-11c1-4baf-82fc-134c519b472a\") " pod="kube-system/konnectivity-agent-4wfbw" Jul 22 00:26:34.219929 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.219875 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:26:34.219929 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.219897 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:26:34.219929 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.219907 2567 projected.go:196] Error preparing data for projected volume kube-api-access-9x68d for pod openshift-network-diagnostics/network-check-target-qtzz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:34.220156 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.219948 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d podName:7c8c37c8-072f-4c0b-84a5-e880dcbb7d17 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:34.719935919 +0000 UTC m=+3.046008101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9x68d" (UniqueName: "kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d") pod "network-check-target-qtzz2" (UID: "7c8c37c8-072f-4c0b-84a5-e880dcbb7d17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:34.221406 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.221378 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45l7k\" (UniqueName: \"kubernetes.io/projected/83fc1f05-76b5-4574-82ad-e12760548aef-kube-api-access-45l7k\") pod \"aws-ebs-csi-driver-node-4hlnw\" (UID: \"83fc1f05-76b5-4574-82ad-e12760548aef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.221520 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.221464 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcclf\" (UniqueName: \"kubernetes.io/projected/07ca19d0-5de0-4391-b15e-2a229d853303-kube-api-access-fcclf\") pod \"ovnkube-node-g5jvn\" (UID: \"07ca19d0-5de0-4391-b15e-2a229d853303\") " pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.221598 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.221574 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhtgk\" (UniqueName: \"kubernetes.io/projected/f46b3a5e-9c2c-4f6e-ab41-0296b66f5255-kube-api-access-rhtgk\") pod \"multus-gvw2z\" (UID: \"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255\") " pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.222056 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.222034 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsm87\" (UniqueName: \"kubernetes.io/projected/d55898f6-dc0e-408a-99cc-9883ae591ac5-kube-api-access-qsm87\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:34.222400 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.222370 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wsfp\" (UniqueName: \"kubernetes.io/projected/9678f689-061e-4efc-94b2-890933759edd-kube-api-access-6wsfp\") pod \"iptables-alerter-pxvv8\" (UID: \"9678f689-061e-4efc-94b2-890933759edd\") " pod="openshift-network-operator/iptables-alerter-pxvv8" Jul 22 00:26:34.292006 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.291974 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5pqtv" Jul 22 00:26:34.301306 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.301283 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lv725" Jul 22 00:26:34.313148 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.313131 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zzww7" Jul 22 00:26:34.319848 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.319829 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kxz6k" Jul 22 00:26:34.327686 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.327668 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gvw2z" Jul 22 00:26:34.337541 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.337526 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:34.346503 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.346484 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pxvv8" Jul 22 00:26:34.355327 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.355305 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4wfbw" Jul 22 00:26:34.362117 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.362099 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" Jul 22 00:26:34.513278 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.513244 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 00:26:34.609078 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.609047 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:34.609218 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.609153 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:34.609218 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.609208 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret podName:30d68a8f-032a-4d0e-90bf-6f70e5b548f2 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:35.609194047 +0000 UTC m=+3.935266227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret") pod "global-pull-secret-syncer-trczs" (UID: "30d68a8f-032a-4d0e-90bf-6f70e5b548f2") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:34.710399 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.710355 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:34.710593 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.710560 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:34.710660 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.710647 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs podName:d55898f6-dc0e-408a-99cc-9883ae591ac5 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:35.710625757 +0000 UTC m=+4.036697964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs") pod "network-metrics-daemon-6q2tc" (UID: "d55898f6-dc0e-408a-99cc-9883ae591ac5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:34.810949 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:34.810869 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x68d\" (UniqueName: \"kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d\") pod \"network-check-target-qtzz2\" (UID: \"7c8c37c8-072f-4c0b-84a5-e880dcbb7d17\") " pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:34.811125 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.811091 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:26:34.811125 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.811118 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:26:34.811231 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.811132 2567 projected.go:196] Error preparing data for projected volume kube-api-access-9x68d for pod openshift-network-diagnostics/network-check-target-qtzz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:34.811231 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:34.811210 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d podName:7c8c37c8-072f-4c0b-84a5-e880dcbb7d17 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:35.811189394 +0000 UTC m=+4.137261581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9x68d" (UniqueName: "kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d") pod "network-check-target-qtzz2" (UID: "7c8c37c8-072f-4c0b-84a5-e880dcbb7d17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:35.008616 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:35.008582 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28ec8cc_d99c_4ca5_9c43_4e0268ec46b5.slice/crio-46c95960681deda12811d4fb301765cd36e756290f5f7a1c9e8649e312d9add3 WatchSource:0}: Error finding container 46c95960681deda12811d4fb301765cd36e756290f5f7a1c9e8649e312d9add3: Status 404 returned error can't find the container with id 46c95960681deda12811d4fb301765cd36e756290f5f7a1c9e8649e312d9add3 Jul 22 00:26:35.009966 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:35.009669 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd30122_7106_4258_af8e_b3c91bf3debe.slice/crio-d8ad04e8b799ec19625e6dab0fc6d480d19c87ca750a8ce2dbc9ab2c4fd91af4 WatchSource:0}: Error finding container d8ad04e8b799ec19625e6dab0fc6d480d19c87ca750a8ce2dbc9ab2c4fd91af4: Status 404 returned error can't find the container with id d8ad04e8b799ec19625e6dab0fc6d480d19c87ca750a8ce2dbc9ab2c4fd91af4 Jul 22 00:26:35.034621 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:35.034599 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d2c403_f38d_4c15_b761_8d50c9f3b142.slice/crio-d48e5f176d286f1aca7de768482044fb15e1c1af0906c57ed28cff84d11a59b0 WatchSource:0}: Error finding container d48e5f176d286f1aca7de768482044fb15e1c1af0906c57ed28cff84d11a59b0: Status 404 returned error can't find the container with id d48e5f176d286f1aca7de768482044fb15e1c1af0906c57ed28cff84d11a59b0 Jul 22 00:26:35.035944 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:35.035924 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60d84ba_a0ae_4e46_9fdf_7830c7d5e798.slice/crio-a4404e90371546bd1afcdcb58180480a323b2c4493abd16e126c26ecc3ac4b62 WatchSource:0}: Error finding container a4404e90371546bd1afcdcb58180480a323b2c4493abd16e126c26ecc3ac4b62: Status 404 returned error can't find the container with id a4404e90371546bd1afcdcb58180480a323b2c4493abd16e126c26ecc3ac4b62 Jul 22 00:26:35.036973 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:35.036942 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46b3a5e_9c2c_4f6e_ab41_0296b66f5255.slice/crio-084f9cf441a8cd23fa5d76cea83b916ad1107783e0d92f7c4a6bcfb3a82261e3 WatchSource:0}: Error finding container 084f9cf441a8cd23fa5d76cea83b916ad1107783e0d92f7c4a6bcfb3a82261e3: Status 404 returned error can't find the container with id 084f9cf441a8cd23fa5d76cea83b916ad1107783e0d92f7c4a6bcfb3a82261e3 Jul 22 00:26:35.039637 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:35.039005 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fc1f05_76b5_4574_82ad_e12760548aef.slice/crio-5ee67fde676a77667f4d1bb6aa00746a1f1409dc94f1b84b6d638be961d85779 WatchSource:0}: Error finding container 5ee67fde676a77667f4d1bb6aa00746a1f1409dc94f1b84b6d638be961d85779: Status 404 returned error can't find the container with id 5ee67fde676a77667f4d1bb6aa00746a1f1409dc94f1b84b6d638be961d85779 Jul 22 00:26:35.041041 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.040991 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 00:21:34 +0000 UTC" deadline="2028-02-27 11:24:07.70345416 +0000 UTC" Jul 22 00:26:35.041041 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.041022 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14050h57m32.662436369s" Jul 22 00:26:35.041718 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:26:35.041622 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9678f689_061e_4efc_94b2_890933759edd.slice/crio-7748ef578988c8edf6b4015e9a1c40448d82df2bd929e1a40a3afbd0c53ca357 WatchSource:0}: Error finding container 7748ef578988c8edf6b4015e9a1c40448d82df2bd929e1a40a3afbd0c53ca357: Status 404 returned error can't find the container with id 7748ef578988c8edf6b4015e9a1c40448d82df2bd929e1a40a3afbd0c53ca357 Jul 22 00:26:35.112065 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.112041 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:35.112177 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:35.112149 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:26:35.122314 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.122275 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" event={"ID":"07ca19d0-5de0-4391-b15e-2a229d853303","Type":"ContainerStarted","Data":"3069b4fedb93e7a02bbe9df7dd7d951b8f285b04d3bf1f596ee12ec0e4da26ed"} Jul 22 00:26:35.123734 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.123701 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pxvv8" event={"ID":"9678f689-061e-4efc-94b2-890933759edd","Type":"ContainerStarted","Data":"7748ef578988c8edf6b4015e9a1c40448d82df2bd929e1a40a3afbd0c53ca357"} Jul 22 00:26:35.124934 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.124910 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" event={"ID":"83fc1f05-76b5-4574-82ad-e12760548aef","Type":"ContainerStarted","Data":"5ee67fde676a77667f4d1bb6aa00746a1f1409dc94f1b84b6d638be961d85779"} Jul 22 00:26:35.126201 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.126174 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gvw2z" event={"ID":"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255","Type":"ContainerStarted","Data":"084f9cf441a8cd23fa5d76cea83b916ad1107783e0d92f7c4a6bcfb3a82261e3"} Jul 22 00:26:35.127587 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.127561 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zzww7" event={"ID":"5dd30122-7106-4258-af8e-b3c91bf3debe","Type":"ContainerStarted","Data":"d8ad04e8b799ec19625e6dab0fc6d480d19c87ca750a8ce2dbc9ab2c4fd91af4"} Jul 22 00:26:35.130736 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.130713 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4wfbw" event={"ID":"f1ef826a-11c1-4baf-82fc-134c519b472a","Type":"ContainerStarted","Data":"31f32cbced78e224c774334bb1b18aa02f41cd31ef3f3717bdb917084a8ddf6c"} Jul 22 00:26:35.131908 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.131885 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lv725" event={"ID":"d60d84ba-a0ae-4e46-9fdf-7830c7d5e798","Type":"ContainerStarted","Data":"a4404e90371546bd1afcdcb58180480a323b2c4493abd16e126c26ecc3ac4b62"} Jul 22 00:26:35.133043 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.133023 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5pqtv" event={"ID":"66d2c403-f38d-4c15-b761-8d50c9f3b142","Type":"ContainerStarted","Data":"d48e5f176d286f1aca7de768482044fb15e1c1af0906c57ed28cff84d11a59b0"} Jul 22 00:26:35.134241 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.134208 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kxz6k" event={"ID":"c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5","Type":"ContainerStarted","Data":"46c95960681deda12811d4fb301765cd36e756290f5f7a1c9e8649e312d9add3"} Jul 22 00:26:35.143599 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.143555 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-178.ec2.internal" podStartSLOduration=2.14354146 podStartE2EDuration="2.14354146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:26:35.143370568 +0000 UTC m=+3.469442773" watchObservedRunningTime="2026-07-22 00:26:35.14354146 +0000 UTC m=+3.469613668" Jul 22 00:26:35.616242 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.615905 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:35.616242 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:35.616149 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:35.616242 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:35.616242 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret podName:30d68a8f-032a-4d0e-90bf-6f70e5b548f2 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:37.616219977 +0000 UTC m=+5.942292162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret") pod "global-pull-secret-syncer-trczs" (UID: "30d68a8f-032a-4d0e-90bf-6f70e5b548f2") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:35.717793 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.717238 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:35.717793 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:35.717408 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:35.717793 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:35.717466 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs podName:d55898f6-dc0e-408a-99cc-9883ae591ac5 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:37.717446281 +0000 UTC m=+6.043518468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs") pod "network-metrics-daemon-6q2tc" (UID: "d55898f6-dc0e-408a-99cc-9883ae591ac5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:35.818544 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:35.818504 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x68d\" (UniqueName: \"kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d\") pod \"network-check-target-qtzz2\" (UID: \"7c8c37c8-072f-4c0b-84a5-e880dcbb7d17\") " pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:35.818702 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:35.818691 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:26:35.818761 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:35.818713 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:26:35.818761 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:35.818728 2567 projected.go:196] Error preparing data for projected volume kube-api-access-9x68d for pod openshift-network-diagnostics/network-check-target-qtzz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:35.818911 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:35.818787 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d podName:7c8c37c8-072f-4c0b-84a5-e880dcbb7d17 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:37.818769017 +0000 UTC m=+6.144841201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9x68d" (UniqueName: "kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d") pod "network-check-target-qtzz2" (UID: "7c8c37c8-072f-4c0b-84a5-e880dcbb7d17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:36.041787 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:36.041725 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 00:21:34 +0000 UTC" deadline="2028-04-13 12:05:46.16725422 +0000 UTC" Jul 22 00:26:36.042420 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:36.041795 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15155h39m10.125463367s" Jul 22 00:26:36.115362 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:36.115282 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:36.115523 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:36.115410 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:26:36.115837 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:36.115812 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:36.115993 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:36.115935 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:26:36.142137 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:36.141266 2567 generic.go:356] "Generic (PLEG): container finished" podID="cbc03ea7b9c0120a2d5366cd2de1b0d7" containerID="d1982d977618871c81ac52bf5a7aaea232a5393e90387ab94675c1f10df32482" exitCode=0 Jul 22 00:26:36.142137 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:36.141906 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal" event={"ID":"cbc03ea7b9c0120a2d5366cd2de1b0d7","Type":"ContainerDied","Data":"d1982d977618871c81ac52bf5a7aaea232a5393e90387ab94675c1f10df32482"} Jul 22 00:26:36.150070 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:36.149571 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-178.ec2.internal" event={"ID":"c36159d4fd5288301a0516429f295afe","Type":"ContainerStarted","Data":"79662ce7abc24b8959347e57337b675097df8ebc50cb5ee227df2b7faeca9dad"} Jul 22 00:26:37.112650 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:37.112620 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:37.113111 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:37.112764 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:26:37.167425 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:37.167391 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal" event={"ID":"cbc03ea7b9c0120a2d5366cd2de1b0d7","Type":"ContainerStarted","Data":"8cd11829ddf983e4e4bedb437148f620ba556289df66863edd8376d3b04de143"} Jul 22 00:26:37.633166 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:37.633076 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:37.633324 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:37.633279 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:37.633380 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:37.633340 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret podName:30d68a8f-032a-4d0e-90bf-6f70e5b548f2 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:41.63332218 +0000 UTC m=+9.959394366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret") pod "global-pull-secret-syncer-trczs" (UID: "30d68a8f-032a-4d0e-90bf-6f70e5b548f2") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:37.734123 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:37.734079 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:37.734384 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:37.734275 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:37.734384 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:37.734340 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs podName:d55898f6-dc0e-408a-99cc-9883ae591ac5 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:41.734321954 +0000 UTC m=+10.060394140 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs") pod "network-metrics-daemon-6q2tc" (UID: "d55898f6-dc0e-408a-99cc-9883ae591ac5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:37.834754 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:37.834715 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x68d\" (UniqueName: \"kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d\") pod \"network-check-target-qtzz2\" (UID: \"7c8c37c8-072f-4c0b-84a5-e880dcbb7d17\") " pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:37.834979 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:37.834944 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:26:37.834979 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:37.834972 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:26:37.835089 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:37.834986 2567 projected.go:196] Error preparing data for projected volume kube-api-access-9x68d for pod openshift-network-diagnostics/network-check-target-qtzz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:37.835089 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:37.835044 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d podName:7c8c37c8-072f-4c0b-84a5-e880dcbb7d17 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:41.835025677 +0000 UTC m=+10.161097876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9x68d" (UniqueName: "kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d") pod "network-check-target-qtzz2" (UID: "7c8c37c8-072f-4c0b-84a5-e880dcbb7d17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:38.112420 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:38.112364 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:38.112606 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:38.112496 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:26:38.112850 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:38.112364 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:38.112850 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:38.112808 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:26:39.113044 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:39.112554 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:39.113044 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:39.112689 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:26:40.112721 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:40.112543 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:40.112721 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:40.112675 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:26:40.113221 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:40.113104 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:40.113221 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:40.113212 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:26:41.113476 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:41.112945 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:41.113476 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:41.113092 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:26:41.667584 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:41.667554 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:41.667804 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:41.667701 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:41.667804 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:41.667762 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret podName:30d68a8f-032a-4d0e-90bf-6f70e5b548f2 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:49.667744329 +0000 UTC m=+17.993816525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret") pod "global-pull-secret-syncer-trczs" (UID: "30d68a8f-032a-4d0e-90bf-6f70e5b548f2") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:41.768650 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:41.768096 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:41.768650 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:41.768266 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:41.768650 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:41.768325 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs podName:d55898f6-dc0e-408a-99cc-9883ae591ac5 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:49.768308593 +0000 UTC m=+18.094380776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs") pod "network-metrics-daemon-6q2tc" (UID: "d55898f6-dc0e-408a-99cc-9883ae591ac5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:41.868760 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:41.868693 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x68d\" (UniqueName: \"kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d\") pod \"network-check-target-qtzz2\" (UID: \"7c8c37c8-072f-4c0b-84a5-e880dcbb7d17\") " pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:41.869377 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:41.868935 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:26:41.869377 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:41.868968 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:26:41.869377 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:41.868983 2567 projected.go:196] Error preparing data for projected volume kube-api-access-9x68d for pod openshift-network-diagnostics/network-check-target-qtzz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:41.869377 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:41.869046 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d podName:7c8c37c8-072f-4c0b-84a5-e880dcbb7d17 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:49.869027212 +0000 UTC m=+18.195099394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9x68d" (UniqueName: "kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d") pod "network-check-target-qtzz2" (UID: "7c8c37c8-072f-4c0b-84a5-e880dcbb7d17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:42.114276 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:42.114245 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:42.114699 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:42.114420 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:26:42.114946 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:42.114928 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:42.115076 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:42.115057 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:26:43.112343 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:43.112266 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:43.112513 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:43.112416 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:26:44.113168 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:44.113135 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:44.113568 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:44.113173 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:44.113568 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:44.113267 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:26:44.113568 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:44.113413 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:26:45.112773 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:45.112739 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:45.112991 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:45.112900 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:26:46.112742 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:46.112708 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:46.113196 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:46.112711 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:46.113196 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:46.112866 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:26:46.113196 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:46.112920 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:26:47.112871 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:47.112823 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:47.113298 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:47.112963 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:26:48.112350 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:48.112307 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:48.112523 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:48.112307 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:48.112523 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:48.112457 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:26:48.112636 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:48.112525 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:26:49.113030 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:49.112989 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:49.113584 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:49.113123 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:26:49.733026 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:49.732979 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:49.733169 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:49.733136 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:49.733210 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:49.733202 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret podName:30d68a8f-032a-4d0e-90bf-6f70e5b548f2 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:05.733186455 +0000 UTC m=+34.059258634 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret") pod "global-pull-secret-syncer-trczs" (UID: "30d68a8f-032a-4d0e-90bf-6f70e5b548f2") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:49.833970 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:49.833933 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:49.834138 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:49.834110 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:49.834204 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:49.834191 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs podName:d55898f6-dc0e-408a-99cc-9883ae591ac5 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:05.834166038 +0000 UTC m=+34.160238246 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs") pod "network-metrics-daemon-6q2tc" (UID: "d55898f6-dc0e-408a-99cc-9883ae591ac5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:49.935029 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:49.934939 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x68d\" (UniqueName: \"kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d\") pod \"network-check-target-qtzz2\" (UID: \"7c8c37c8-072f-4c0b-84a5-e880dcbb7d17\") " pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:49.935181 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:49.935096 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:26:49.935181 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:49.935115 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:26:49.935181 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:49.935124 2567 projected.go:196] Error preparing data for projected volume kube-api-access-9x68d for pod openshift-network-diagnostics/network-check-target-qtzz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:49.935181 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:49.935172 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d podName:7c8c37c8-072f-4c0b-84a5-e880dcbb7d17 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:05.935159633 +0000 UTC m=+34.261231817 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9x68d" (UniqueName: "kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d") pod "network-check-target-qtzz2" (UID: "7c8c37c8-072f-4c0b-84a5-e880dcbb7d17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:50.115635 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:50.115605 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:50.116054 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:50.115708 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:26:50.116054 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:50.115766 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:50.116054 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:50.115883 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:26:51.112583 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:51.112551 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:51.112800 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:51.112697 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:26:52.115458 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:52.115424 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:52.115969 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:52.115435 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:52.115969 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:52.115549 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:26:52.115969 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:52.115654 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:26:53.112691 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:53.112658 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:53.112872 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:53.112800 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:26:54.115205 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.115026 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:54.115566 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.115049 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:54.115566 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:54.115383 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:26:54.115566 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:54.115278 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:26:54.202217 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.202188 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kxz6k" event={"ID":"c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5","Type":"ContainerStarted","Data":"03d31412666dfdaae884e8a7b9626b9c2951c3735a8c5385a36de7594c36f8ff"} Jul 22 00:26:54.204044 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.204027 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:26:54.204364 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.204341 2567 generic.go:356] "Generic (PLEG): container finished" podID="07ca19d0-5de0-4391-b15e-2a229d853303" containerID="0a910fd86deeb68e130a2f5f1469b20e86664e2d2c9ad5cb9aba65073895d5a1" exitCode=1 Jul 22 00:26:54.204460 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.204368 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" event={"ID":"07ca19d0-5de0-4391-b15e-2a229d853303","Type":"ContainerStarted","Data":"c897f1c983b03347b65eec37fd64534822927831d2d24c10ab2e4752a68307ca"} Jul 22 00:26:54.204460 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.204455 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" event={"ID":"07ca19d0-5de0-4391-b15e-2a229d853303","Type":"ContainerDied","Data":"0a910fd86deeb68e130a2f5f1469b20e86664e2d2c9ad5cb9aba65073895d5a1"} Jul 22 00:26:54.204833 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.204509 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" event={"ID":"07ca19d0-5de0-4391-b15e-2a229d853303","Type":"ContainerStarted","Data":"b6004b8ea2022ce098bb7f2f835e8370aa72be24aa6b7cd0f5855e88f06971af"} Jul 22 00:26:54.205899 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.205867 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" event={"ID":"83fc1f05-76b5-4574-82ad-e12760548aef","Type":"ContainerStarted","Data":"51cf15a2c87da323b3255f87699591e544417f0070a5157ef1ed6f67b749be84"} Jul 22 00:26:54.208017 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.207990 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gvw2z" event={"ID":"f46b3a5e-9c2c-4f6e-ab41-0296b66f5255","Type":"ContainerStarted","Data":"ed67ca763f3ad6a955a7523b80fe850270bf470b1cdbd287ec6ba3fc05cd72e7"} Jul 22 00:26:54.209362 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.209339 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zzww7" event={"ID":"5dd30122-7106-4258-af8e-b3c91bf3debe","Type":"ContainerStarted","Data":"4af14de2efac631fb9613e3aadcc67b22d7739ce51d31a030d196fffc2625d64"} Jul 22 00:26:54.210511 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.210494 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4wfbw" event={"ID":"f1ef826a-11c1-4baf-82fc-134c519b472a","Type":"ContainerStarted","Data":"1db93bb60fcdd21b3b97af350961a1128ae8bbecfe1a446e24b5f5e7b0d99554"} Jul 22 00:26:54.211688 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.211669 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lv725" event={"ID":"d60d84ba-a0ae-4e46-9fdf-7830c7d5e798","Type":"ContainerStarted","Data":"58b1499795fa4902d1c1ee28865cbbb34c5a8236b3ee6bc0879ff86e31f8a872"} Jul 22 00:26:54.213116 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.213097 2567 generic.go:356] "Generic (PLEG): container finished" podID="66d2c403-f38d-4c15-b761-8d50c9f3b142" containerID="b32af2b195ad137b0648a6526cfa28a7fb4f8db8f3ec492e0c87cc349ecfa681" exitCode=0 Jul 22 00:26:54.213207 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.213128 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5pqtv" event={"ID":"66d2c403-f38d-4c15-b761-8d50c9f3b142","Type":"ContainerDied","Data":"b32af2b195ad137b0648a6526cfa28a7fb4f8db8f3ec492e0c87cc349ecfa681"} Jul 22 00:26:54.216909 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.216873 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kxz6k" podStartSLOduration=3.858930242 podStartE2EDuration="22.216861946s" totalImagesPullingTime="18.357931704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:26:54.216478885 +0000 UTC m=+22.542551087" watchObservedRunningTime="2026-07-22 00:26:54.216861946 +0000 UTC m=+22.542934147" Jul 22 00:26:54.216994 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.216937 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-178.ec2.internal" podStartSLOduration=21.216932324 podStartE2EDuration="21.216932324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:26:37.183243856 +0000 UTC m=+5.509316059" watchObservedRunningTime="2026-07-22 00:26:54.216932324 +0000 UTC m=+22.543004525" Jul 22 00:26:54.249630 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.249421 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lv725" podStartSLOduration=3.855078128 podStartE2EDuration="22.249400745s" totalImagesPullingTime="18.394322617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:26:54.24879951 +0000 UTC m=+22.574871706" watchObservedRunningTime="2026-07-22 00:26:54.249400745 +0000 UTC m=+22.575472948" Jul 22 00:26:54.265294 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.265252 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gvw2z" podStartSLOduration=3.488580809 podStartE2EDuration="22.265236379s" totalImagesPullingTime="18.77665557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:26:54.264714084 +0000 UTC m=+22.590786286" watchObservedRunningTime="2026-07-22 00:26:54.265236379 +0000 UTC m=+22.591308581" Jul 22 00:26:54.288078 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.288035 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4wfbw" podStartSLOduration=3.9405787180000003 podStartE2EDuration="22.288020418s" totalImagesPullingTime="18.3474417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:26:54.287765513 +0000 UTC m=+22.613837718" watchObservedRunningTime="2026-07-22 00:26:54.288020418 +0000 UTC m=+22.614092620" Jul 22 00:26:54.306657 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:54.306617 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zzww7" podStartSLOduration=3.903285866 podStartE2EDuration="22.306600014s" totalImagesPullingTime="18.403314148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:26:54.306209644 +0000 UTC m=+22.632281846" watchObservedRunningTime="2026-07-22 00:26:54.306600014 +0000 UTC m=+22.632672215" Jul 22 00:26:55.112844 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:55.112810 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:55.113070 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:55.112924 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:26:55.216358 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:55.216319 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pxvv8" event={"ID":"9678f689-061e-4efc-94b2-890933759edd","Type":"ContainerStarted","Data":"9364c11d6d85b07e30fbe1b2f1ac4fa0f8934bc7f2837cfd78b10482231f0166"} Jul 22 00:26:55.219008 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:55.218988 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:26:55.219345 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:55.219323 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" event={"ID":"07ca19d0-5de0-4391-b15e-2a229d853303","Type":"ContainerStarted","Data":"cfb9578665a3bd825179659aed86c16d9af0bbf5cc1876c7df64d92ff4519abd"} Jul 22 00:26:55.219424 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:55.219405 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" event={"ID":"07ca19d0-5de0-4391-b15e-2a229d853303","Type":"ContainerStarted","Data":"a24098bcb521869f44415337fb1619b08761a8c4a25ccb1c534d840d85fae300"} Jul 22 00:26:55.219480 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:55.219443 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" event={"ID":"07ca19d0-5de0-4391-b15e-2a229d853303","Type":"ContainerStarted","Data":"59a600205a581ee73dfdc5a00932a21700b6bc0b0c24143b9459f32ad4d2a2fd"} Jul 22 00:26:55.235412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:55.235361 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pxvv8" podStartSLOduration=4.890151868 podStartE2EDuration="23.235346068s" totalImagesPullingTime="18.3451942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:26:55.234722451 +0000 UTC m=+23.560794653" watchObservedRunningTime="2026-07-22 00:26:55.235346068 +0000 UTC m=+23.561418270" Jul 22 00:26:55.411595 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:55.411567 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 00:26:56.023234 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:56.023199 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4wfbw" Jul 22 00:26:56.023917 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:56.023894 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4wfbw" Jul 22 00:26:56.059207 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:56.059079 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T00:26:55.411587578Z","UUID":"fdee7397-dbc6-48da-b7d3-3b3bb6e18516","Handler":null,"Name":"","Endpoint":""} Jul 22 00:26:56.060709 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:56.060689 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 00:26:56.060851 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:56.060740 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 00:26:56.115219 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:56.115191 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:56.115389 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:56.115231 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:56.115389 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:56.115304 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:26:56.115474 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:56.115406 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:26:56.223087 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:56.223053 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" event={"ID":"83fc1f05-76b5-4574-82ad-e12760548aef","Type":"ContainerStarted","Data":"48edc643771ce2ec493e5de5ededcc396bad752384afec5bfe46170a0332a0de"} Jul 22 00:26:57.112244 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:57.112209 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:57.112432 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:57.112338 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:26:57.228514 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:57.228476 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:26:57.228964 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:57.228935 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" event={"ID":"07ca19d0-5de0-4391-b15e-2a229d853303","Type":"ContainerStarted","Data":"266671157ad09a847b5a236b6f68c85d56342a479882b38e0a763af6e5efe801"} Jul 22 00:26:57.231155 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:57.231132 2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 22 00:26:57.231155 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:57.231131 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" event={"ID":"83fc1f05-76b5-4574-82ad-e12760548aef","Type":"ContainerStarted","Data":"48a58bd32a20244ab88a824a95105d89b6ac40f57409860cc783643ba7cfb463"} Jul 22 00:26:57.248519 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:57.248474 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hlnw" podStartSLOduration=4.129139173 podStartE2EDuration="25.248458097s" totalImagesPullingTime="21.119318924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:26:57.247920213 +0000 UTC m=+25.573992415" watchObservedRunningTime="2026-07-22 00:26:57.248458097 +0000 UTC m=+25.574530310" Jul 22 00:26:58.115280 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:58.115247 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:26:58.115457 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:58.115249 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:26:58.115457 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:58.115366 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:26:58.115457 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:58.115417 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:26:59.112219 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:59.111988 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:26:59.112851 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:26:59.112332 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:26:59.239961 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:59.239931 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:26:59.240347 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:59.240321 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" event={"ID":"07ca19d0-5de0-4391-b15e-2a229d853303","Type":"ContainerStarted","Data":"f54eecb911052fda7449540f6366313aec7034c125c52108ce12a0ad15aa1a92"} Jul 22 00:26:59.240628 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:59.240597 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:26:59.240799 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:59.240769 2567 scope.go:117] "RemoveContainer" containerID="0a910fd86deeb68e130a2f5f1469b20e86664e2d2c9ad5cb9aba65073895d5a1" Jul 22 00:26:59.257223 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:26:59.257189 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:27:00.112360 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:00.112329 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:27:00.112955 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:00.112441 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:27:00.112955 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:00.112329 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:27:00.112955 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:00.112888 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:27:00.513635 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:00.513607 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4wfbw" Jul 22 00:27:00.513803 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:00.513740 2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 22 00:27:00.514202 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:00.514178 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4wfbw" Jul 22 00:27:01.009391 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:01.009187 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-trczs"] Jul 22 00:27:01.009521 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:01.009473 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:27:01.009598 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:01.009581 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:27:01.012488 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:01.012457 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6q2tc"] Jul 22 00:27:01.012646 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:01.012586 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:27:01.012711 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:01.012683 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:27:01.013098 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:01.013064 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qtzz2"] Jul 22 00:27:01.013198 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:01.013145 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:27:01.013255 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:01.013233 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:27:01.247034 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:01.246959 2567 generic.go:356] "Generic (PLEG): container finished" podID="66d2c403-f38d-4c15-b761-8d50c9f3b142" containerID="9d8ad4c41751ae2dbb7f1525d918e13aa5ba023c62e2e4693cf6fbbb840a03a4" exitCode=0 Jul 22 00:27:01.247424 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:01.247041 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5pqtv" event={"ID":"66d2c403-f38d-4c15-b761-8d50c9f3b142","Type":"ContainerDied","Data":"9d8ad4c41751ae2dbb7f1525d918e13aa5ba023c62e2e4693cf6fbbb840a03a4"} Jul 22 00:27:01.250960 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:01.250945 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:27:01.251352 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:01.251328 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" event={"ID":"07ca19d0-5de0-4391-b15e-2a229d853303","Type":"ContainerStarted","Data":"4c10c4855a7c77924e6cf230366938204dad6359cb37304e233669fc720c2aaa"} Jul 22 00:27:01.251446 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:01.251416 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:27:01.251606 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:01.251590 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:27:01.266531 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:01.266496 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:27:01.303380 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:01.303326 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" podStartSLOduration=10.394987411 podStartE2EDuration="29.303311323s" totalImagesPullingTime="18.908323912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:01.303087319 +0000 UTC m=+29.629159520" watchObservedRunningTime="2026-07-22 00:27:01.303311323 +0000 UTC m=+29.629383517" Jul 22 00:27:03.112556 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:03.112514 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:27:03.112943 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:03.112571 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:27:03.112943 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:03.112522 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:27:03.112943 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:03.112632 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:27:03.112943 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:03.112685 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:27:03.112943 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:03.112764 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:27:03.258279 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:03.258249 2567 generic.go:356] "Generic (PLEG): container finished" podID="66d2c403-f38d-4c15-b761-8d50c9f3b142" containerID="7b6ea52fab7f84bb4912851eafe85a46d8534e4b2e4cc55b7a16daf6b21ec195" exitCode=0 Jul 22 00:27:03.258447 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:03.258332 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5pqtv" event={"ID":"66d2c403-f38d-4c15-b761-8d50c9f3b142","Type":"ContainerDied","Data":"7b6ea52fab7f84bb4912851eafe85a46d8534e4b2e4cc55b7a16daf6b21ec195"} Jul 22 00:27:03.274065 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:03.274029 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" podUID="07ca19d0-5de0-4391-b15e-2a229d853303" containerName="ovnkube-controller" probeResult="failure" output="" Jul 22 00:27:05.113188 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:05.113000 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:27:05.113600 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:05.113034 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:27:05.113600 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:05.113249 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:27:05.113600 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:05.113365 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:27:05.113600 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:05.113054 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:27:05.113600 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:05.113459 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:27:05.264325 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:05.264295 2567 generic.go:356] "Generic (PLEG): container finished" podID="66d2c403-f38d-4c15-b761-8d50c9f3b142" containerID="acbacb0e046230f2cff4d7d2b71b71a3260e7ffc9143050ee25b0e45beb50da8" exitCode=0 Jul 22 00:27:05.264460 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:05.264347 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5pqtv" event={"ID":"66d2c403-f38d-4c15-b761-8d50c9f3b142","Type":"ContainerDied","Data":"acbacb0e046230f2cff4d7d2b71b71a3260e7ffc9143050ee25b0e45beb50da8"} Jul 22 00:27:05.753268 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:05.753241 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:27:05.753419 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:05.753349 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:05.753419 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:05.753395 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret podName:30d68a8f-032a-4d0e-90bf-6f70e5b548f2 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:37.753383013 +0000 UTC m=+66.079455206 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret") pod "global-pull-secret-syncer-trczs" (UID: "30d68a8f-032a-4d0e-90bf-6f70e5b548f2") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:05.854352 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:05.854320 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:27:05.854484 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:05.854414 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:05.854484 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:05.854462 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs podName:d55898f6-dc0e-408a-99cc-9883ae591ac5 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:37.854448873 +0000 UTC m=+66.180521054 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs") pod "network-metrics-daemon-6q2tc" (UID: "d55898f6-dc0e-408a-99cc-9883ae591ac5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:05.955296 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:05.955268 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x68d\" (UniqueName: \"kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d\") pod \"network-check-target-qtzz2\" (UID: \"7c8c37c8-072f-4c0b-84a5-e880dcbb7d17\") " pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:27:05.955445 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:05.955381 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:27:05.955445 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:05.955395 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:27:05.955445 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:05.955403 2567 projected.go:196] Error preparing data for projected volume kube-api-access-9x68d for pod openshift-network-diagnostics/network-check-target-qtzz2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:05.955445 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:05.955446 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d podName:7c8c37c8-072f-4c0b-84a5-e880dcbb7d17 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:37.955433142 +0000 UTC m=+66.281505321 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9x68d" (UniqueName: "kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d") pod "network-check-target-qtzz2" (UID: "7c8c37c8-072f-4c0b-84a5-e880dcbb7d17") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:07.112825 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.112793 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:27:07.112825 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.112818 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:27:07.113329 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.112818 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:27:07.113329 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:07.112933 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:27:07.113329 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:07.113060 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtzz2" podUID="7c8c37c8-072f-4c0b-84a5-e880dcbb7d17" Jul 22 00:27:07.113329 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:07.113144 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trczs" podUID="30d68a8f-032a-4d0e-90bf-6f70e5b548f2" Jul 22 00:27:07.436850 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.436766 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-178.ec2.internal" event="NodeReady" Jul 22 00:27:07.437010 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.436918 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 00:27:07.469334 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.469303 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr"] Jul 22 00:27:07.487515 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.487477 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58fcb4cbcd-j5fmf"] Jul 22 00:27:07.487680 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.487635 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:27:07.494006 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.491742 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 22 00:27:07.495540 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.495519 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 22 00:27:07.495656 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.495516 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-45lzm\"" Jul 22 00:27:07.499613 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.499594 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr"] Jul 22 00:27:07.499722 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.499630 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cdzlx"] Jul 22 00:27:07.499791 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.499717 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.502469 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.502444 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 00:27:07.502574 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.502470 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 00:27:07.502574 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.502522 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 00:27:07.502574 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.502569 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kstbg\"" Jul 22 00:27:07.509357 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.509337 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 00:27:07.514178 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.514158 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58fcb4cbcd-j5fmf"] Jul 22 00:27:07.514270 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.514195 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5kzgh"] Jul 22 00:27:07.514332 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.514278 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:07.516652 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.516634 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nfkpn\"" Jul 22 00:27:07.516810 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.516768 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 00:27:07.516810 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.516635 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 00:27:07.532483 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.532461 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cdzlx"] Jul 22 00:27:07.532574 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.532499 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5kzgh"] Jul 22 00:27:07.532623 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.532604 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5kzgh" Jul 22 00:27:07.534842 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.534822 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9frz8\"" Jul 22 00:27:07.535222 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.535117 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 00:27:07.535222 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.535118 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 00:27:07.535222 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.535150 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 00:27:07.566522 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.566489 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-zmtpr\" (UID: \"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:27:07.566680 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.566534 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-zmtpr\" (UID: \"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:27:07.667886 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.667844 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.667886 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.667892 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a-cert\") pod \"ingress-canary-5kzgh\" (UID: \"2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a\") " pod="openshift-ingress-canary/ingress-canary-5kzgh" Jul 22 00:27:07.668130 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.667920 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0a1eb7a-b0dc-4804-975f-c601526d229e-image-registry-private-configuration\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.668130 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.667944 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-bound-sa-token\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.668130 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.668009 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:07.668130 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.668057 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0a1eb7a-b0dc-4804-975f-c601526d229e-ca-trust-extracted\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.668130 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.668103 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-zmtpr\" (UID: \"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:27:07.668376 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.668153 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kqz4\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-kube-api-access-8kqz4\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.668376 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.668183 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-certificates\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.668376 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:07.668194 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 00:27:07.668376 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.668207 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f69f381-426d-422c-a9df-a007c2e4d0e6-config-volume\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:07.668376 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.668231 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f69f381-426d-422c-a9df-a007c2e4d0e6-tmp-dir\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:07.668376 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:07.668251 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert podName:e97ba97d-b165-4bd4-9a46-d8ac502f5d9d nodeName:}" failed. No retries permitted until 2026-07-22 00:27:08.168231353 +0000 UTC m=+36.494303533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-zmtpr" (UID: "e97ba97d-b165-4bd4-9a46-d8ac502f5d9d") : secret "networking-console-plugin-cert" not found Jul 22 00:27:07.668376 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.668291 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-zmtpr\" (UID: \"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:27:07.668376 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.668318 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8t78\" (UniqueName: \"kubernetes.io/projected/0f69f381-426d-422c-a9df-a007c2e4d0e6-kube-api-access-m8t78\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:07.668376 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.668345 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0a1eb7a-b0dc-4804-975f-c601526d229e-installation-pull-secrets\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.668376 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.668370 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d42ll\" (UniqueName: \"kubernetes.io/projected/2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a-kube-api-access-d42ll\") pod \"ingress-canary-5kzgh\" (UID: \"2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a\") " pod="openshift-ingress-canary/ingress-canary-5kzgh" Jul 22 00:27:07.668762 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.668476 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0a1eb7a-b0dc-4804-975f-c601526d229e-trusted-ca\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.669056 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.669035 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-zmtpr\" (UID: \"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:27:07.769084 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769051 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-certificates\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.769278 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769094 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f69f381-426d-422c-a9df-a007c2e4d0e6-config-volume\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:07.769278 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769121 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f69f381-426d-422c-a9df-a007c2e4d0e6-tmp-dir\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:07.769278 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769150 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8t78\" (UniqueName: \"kubernetes.io/projected/0f69f381-426d-422c-a9df-a007c2e4d0e6-kube-api-access-m8t78\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:07.769278 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769180 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0a1eb7a-b0dc-4804-975f-c601526d229e-installation-pull-secrets\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.769278 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769207 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d42ll\" (UniqueName: \"kubernetes.io/projected/2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a-kube-api-access-d42ll\") pod \"ingress-canary-5kzgh\" (UID: \"2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a\") " pod="openshift-ingress-canary/ingress-canary-5kzgh" Jul 22 00:27:07.769278 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769259 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0a1eb7a-b0dc-4804-975f-c601526d229e-trusted-ca\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.769572 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769316 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.769572 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769356 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a-cert\") pod \"ingress-canary-5kzgh\" (UID: \"2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a\") " pod="openshift-ingress-canary/ingress-canary-5kzgh" Jul 22 00:27:07.769572 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769385 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0a1eb7a-b0dc-4804-975f-c601526d229e-image-registry-private-configuration\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.769572 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769412 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-bound-sa-token\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.769572 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769438 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:07.769572 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769465 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0a1eb7a-b0dc-4804-975f-c601526d229e-ca-trust-extracted\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.769572 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769512 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f69f381-426d-422c-a9df-a007c2e4d0e6-tmp-dir\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:07.769572 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769521 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kqz4\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-kube-api-access-8kqz4\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.769946 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:07.769657 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:07.769946 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:07.769723 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls podName:0f69f381-426d-422c-a9df-a007c2e4d0e6 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:08.269706202 +0000 UTC m=+36.595778383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls") pod "dns-default-cdzlx" (UID: "0f69f381-426d-422c-a9df-a007c2e4d0e6") : secret "dns-default-metrics-tls" not found Jul 22 00:27:07.769946 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769732 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-certificates\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.769946 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.769743 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f69f381-426d-422c-a9df-a007c2e4d0e6-config-volume\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:07.770126 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.770034 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0a1eb7a-b0dc-4804-975f-c601526d229e-ca-trust-extracted\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.770174 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:07.770124 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 00:27:07.770174 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:07.770138 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58fcb4cbcd-j5fmf: secret "image-registry-tls" not found Jul 22 00:27:07.770271 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:07.770196 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls podName:e0a1eb7a-b0dc-4804-975f-c601526d229e nodeName:}" failed. No retries permitted until 2026-07-22 00:27:08.270178091 +0000 UTC m=+36.596250276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls") pod "image-registry-58fcb4cbcd-j5fmf" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e") : secret "image-registry-tls" not found Jul 22 00:27:07.770988 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.770752 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0a1eb7a-b0dc-4804-975f-c601526d229e-trusted-ca\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.774354 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.774331 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0a1eb7a-b0dc-4804-975f-c601526d229e-image-registry-private-configuration\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.774354 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.774338 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a-cert\") pod \"ingress-canary-5kzgh\" (UID: \"2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a\") " pod="openshift-ingress-canary/ingress-canary-5kzgh" Jul 22 00:27:07.774525 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.774338 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0a1eb7a-b0dc-4804-975f-c601526d229e-installation-pull-secrets\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.778692 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.778668 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d42ll\" (UniqueName: \"kubernetes.io/projected/2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a-kube-api-access-d42ll\") pod \"ingress-canary-5kzgh\" (UID: \"2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a\") " pod="openshift-ingress-canary/ingress-canary-5kzgh" Jul 22 00:27:07.778837 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.778816 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kqz4\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-kube-api-access-8kqz4\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.779152 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.779132 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8t78\" (UniqueName: \"kubernetes.io/projected/0f69f381-426d-422c-a9df-a007c2e4d0e6-kube-api-access-m8t78\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:07.779491 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.779467 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-bound-sa-token\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:07.842015 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:07.841978 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5kzgh" Jul 22 00:27:08.067331 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:08.067241 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5kzgh"] Jul 22 00:27:08.071476 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:27:08.071445 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f80c51d_33fe_4bf1_8a2d_e6d0f2f0982a.slice/crio-390ecf900fb2e8f7dbef6b7454eabba3dd1d94314dfa0ce971f91bf8506cdb06 WatchSource:0}: Error finding container 390ecf900fb2e8f7dbef6b7454eabba3dd1d94314dfa0ce971f91bf8506cdb06: Status 404 returned error can't find the container with id 390ecf900fb2e8f7dbef6b7454eabba3dd1d94314dfa0ce971f91bf8506cdb06 Jul 22 00:27:08.173825 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:08.173768 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-zmtpr\" (UID: \"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:27:08.174604 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:08.173929 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 00:27:08.174604 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:08.174013 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert podName:e97ba97d-b165-4bd4-9a46-d8ac502f5d9d nodeName:}" failed. No retries permitted until 2026-07-22 00:27:09.173994594 +0000 UTC m=+37.500066789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-zmtpr" (UID: "e97ba97d-b165-4bd4-9a46-d8ac502f5d9d") : secret "networking-console-plugin-cert" not found Jul 22 00:27:08.271583 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:08.271414 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5kzgh" event={"ID":"2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a","Type":"ContainerStarted","Data":"390ecf900fb2e8f7dbef6b7454eabba3dd1d94314dfa0ce971f91bf8506cdb06"} Jul 22 00:27:08.277880 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:08.277847 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:08.278231 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:08.278201 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:08.278363 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:08.278242 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 00:27:08.278363 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:08.278258 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58fcb4cbcd-j5fmf: secret "image-registry-tls" not found Jul 22 00:27:08.278363 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:08.278302 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:08.278363 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:08.278319 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls podName:e0a1eb7a-b0dc-4804-975f-c601526d229e nodeName:}" failed. No retries permitted until 2026-07-22 00:27:09.27830043 +0000 UTC m=+37.604372624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls") pod "image-registry-58fcb4cbcd-j5fmf" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e") : secret "image-registry-tls" not found Jul 22 00:27:08.278363 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:08.278362 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls podName:0f69f381-426d-422c-a9df-a007c2e4d0e6 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:09.278344118 +0000 UTC m=+37.604416298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls") pod "dns-default-cdzlx" (UID: "0f69f381-426d-422c-a9df-a007c2e4d0e6") : secret "dns-default-metrics-tls" not found Jul 22 00:27:09.112477 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:09.112396 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:27:09.112477 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:09.112439 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:27:09.112707 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:09.112511 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:27:09.116329 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:09.116172 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 00:27:09.116329 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:09.116202 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 00:27:09.116329 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:09.116222 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 00:27:09.116329 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:09.116239 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rsxjh\"" Jul 22 00:27:09.116605 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:09.116344 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 00:27:09.116605 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:09.116505 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dh9z6\"" Jul 22 00:27:09.187223 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:09.187152 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-zmtpr\" (UID: \"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:27:09.187681 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:09.187330 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 00:27:09.187681 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:09.187404 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert podName:e97ba97d-b165-4bd4-9a46-d8ac502f5d9d nodeName:}" failed. No retries permitted until 2026-07-22 00:27:11.187380494 +0000 UTC m=+39.513452679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-zmtpr" (UID: "e97ba97d-b165-4bd4-9a46-d8ac502f5d9d") : secret "networking-console-plugin-cert" not found Jul 22 00:27:09.287675 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:09.287635 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:09.287890 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:09.287710 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:09.287890 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:09.287844 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 00:27:09.287890 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:09.287866 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58fcb4cbcd-j5fmf: secret "image-registry-tls" not found Jul 22 00:27:09.287890 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:09.287884 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:09.288078 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:09.287943 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls podName:e0a1eb7a-b0dc-4804-975f-c601526d229e nodeName:}" failed. No retries permitted until 2026-07-22 00:27:11.287922395 +0000 UTC m=+39.613994597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls") pod "image-registry-58fcb4cbcd-j5fmf" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e") : secret "image-registry-tls" not found Jul 22 00:27:09.288078 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:09.287964 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls podName:0f69f381-426d-422c-a9df-a007c2e4d0e6 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:11.287953675 +0000 UTC m=+39.614025854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls") pod "dns-default-cdzlx" (UID: "0f69f381-426d-422c-a9df-a007c2e4d0e6") : secret "dns-default-metrics-tls" not found Jul 22 00:27:11.205679 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:11.205645 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-zmtpr\" (UID: \"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:27:11.206104 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:11.205823 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 00:27:11.206104 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:11.205906 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert podName:e97ba97d-b165-4bd4-9a46-d8ac502f5d9d nodeName:}" failed. No retries permitted until 2026-07-22 00:27:15.20588418 +0000 UTC m=+43.531956383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-zmtpr" (UID: "e97ba97d-b165-4bd4-9a46-d8ac502f5d9d") : secret "networking-console-plugin-cert" not found Jul 22 00:27:11.306297 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:11.306264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:11.306462 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:11.306396 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:11.306462 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:11.306436 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:11.306543 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:11.306492 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 00:27:11.306543 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:11.306504 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58fcb4cbcd-j5fmf: secret "image-registry-tls" not found Jul 22 00:27:11.306543 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:11.306509 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls podName:0f69f381-426d-422c-a9df-a007c2e4d0e6 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:15.306490346 +0000 UTC m=+43.632562541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls") pod "dns-default-cdzlx" (UID: "0f69f381-426d-422c-a9df-a007c2e4d0e6") : secret "dns-default-metrics-tls" not found Jul 22 00:27:11.306543 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:11.306536 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls podName:e0a1eb7a-b0dc-4804-975f-c601526d229e nodeName:}" failed. No retries permitted until 2026-07-22 00:27:15.306527012 +0000 UTC m=+43.632599210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls") pod "image-registry-58fcb4cbcd-j5fmf" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e") : secret "image-registry-tls" not found Jul 22 00:27:11.602812 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:11.602762 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:11.684728 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:11.684690 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:12.283854 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:12.283815 2567 generic.go:356] "Generic (PLEG): container finished" podID="66d2c403-f38d-4c15-b761-8d50c9f3b142" containerID="53ab03c58f3c6e35fb99f72146c342a91db521ad11b6f00ed3b199f4d3fb3487" exitCode=0 Jul 22 00:27:12.284211 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:12.283897 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5pqtv" event={"ID":"66d2c403-f38d-4c15-b761-8d50c9f3b142","Type":"ContainerDied","Data":"53ab03c58f3c6e35fb99f72146c342a91db521ad11b6f00ed3b199f4d3fb3487"} Jul 22 00:27:12.285440 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:12.285398 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5kzgh" event={"ID":"2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a","Type":"ContainerStarted","Data":"296e837390c53ceb8f0e0715a33ca2be77ca663f40697979337cba346ed4f33f"} Jul 22 00:27:12.317663 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:12.317611 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5kzgh" podStartSLOduration=1.7129399699999999 podStartE2EDuration="5.317593779s" totalImagesPullingTime="3.604653809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:12.31676474 +0000 UTC m=+40.642836967" watchObservedRunningTime="2026-07-22 00:27:12.317593779 +0000 UTC m=+40.643665981" Jul 22 00:27:13.290550 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:13.290514 2567 generic.go:356] "Generic (PLEG): container finished" podID="66d2c403-f38d-4c15-b761-8d50c9f3b142" containerID="f45c62d4ac504527a03a8cd47d284369ae5948acf3c4d43f2e0336b1567c7f35" exitCode=0 Jul 22 00:27:13.291029 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:13.290577 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5pqtv" event={"ID":"66d2c403-f38d-4c15-b761-8d50c9f3b142","Type":"ContainerDied","Data":"f45c62d4ac504527a03a8cd47d284369ae5948acf3c4d43f2e0336b1567c7f35"} Jul 22 00:27:13.426866 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:13.426836 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:13.484703 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:13.484669 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:14.297193 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:14.297151 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5pqtv" event={"ID":"66d2c403-f38d-4c15-b761-8d50c9f3b142","Type":"ContainerStarted","Data":"8a23c309e812f71ba85a7a9f14162b9da3371ded9d8b76d6f5055275932471fa"} Jul 22 00:27:14.319170 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:14.319114 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5pqtv" podStartSLOduration=8.022513194 podStartE2EDuration="42.319101777s" totalImagesPullingTime="34.296588583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:14.317342198 +0000 UTC m=+42.643414399" watchObservedRunningTime="2026-07-22 00:27:14.319101777 +0000 UTC m=+42.645173978" Jul 22 00:27:14.437553 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:14.437519 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:14.486768 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:14.486730 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:15.238239 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:15.238197 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-zmtpr\" (UID: \"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:27:15.238409 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:15.238355 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 00:27:15.238446 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:15.238426 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert podName:e97ba97d-b165-4bd4-9a46-d8ac502f5d9d nodeName:}" failed. No retries permitted until 2026-07-22 00:27:23.238409335 +0000 UTC m=+51.564481518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-zmtpr" (UID: "e97ba97d-b165-4bd4-9a46-d8ac502f5d9d") : secret "networking-console-plugin-cert" not found Jul 22 00:27:15.338845 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:15.338809 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:15.339283 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:15.338857 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:15.339283 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:15.338965 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 00:27:15.339283 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:15.338983 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58fcb4cbcd-j5fmf: secret "image-registry-tls" not found Jul 22 00:27:15.339283 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:15.338983 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:15.339283 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:15.339036 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls podName:e0a1eb7a-b0dc-4804-975f-c601526d229e nodeName:}" failed. No retries permitted until 2026-07-22 00:27:23.339021441 +0000 UTC m=+51.665093622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls") pod "image-registry-58fcb4cbcd-j5fmf" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e") : secret "image-registry-tls" not found Jul 22 00:27:15.339283 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:15.339051 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls podName:0f69f381-426d-422c-a9df-a007c2e4d0e6 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:23.339043813 +0000 UTC m=+51.665115993 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls") pod "dns-default-cdzlx" (UID: "0f69f381-426d-422c-a9df-a007c2e4d0e6") : secret "dns-default-metrics-tls" not found Jul 22 00:27:17.321943 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:17.321902 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:17.372525 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:17.372492 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:18.187360 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:18.187324 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:18.270050 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:18.270018 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:18.323111 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:18.323082 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:18.662215 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:18.662177 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:18.762205 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:18.762170 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:19.096032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:19.095992 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:19.477029 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:19.476945 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:19.862050 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:19.862015 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:19.961576 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:19.961540 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:21.251913 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:21.251879 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:23.297238 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:23.297194 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-zmtpr\" (UID: \"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:27:23.297690 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:23.297362 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 00:27:23.297690 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:23.297434 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert podName:e97ba97d-b165-4bd4-9a46-d8ac502f5d9d nodeName:}" failed. No retries permitted until 2026-07-22 00:27:39.297417894 +0000 UTC m=+67.623490088 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-zmtpr" (UID: "e97ba97d-b165-4bd4-9a46-d8ac502f5d9d") : secret "networking-console-plugin-cert" not found Jul 22 00:27:23.398326 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:23.398286 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:23.398326 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:23.398330 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:23.398590 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:23.398444 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:23.398590 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:23.398452 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 00:27:23.398590 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:23.398473 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58fcb4cbcd-j5fmf: secret "image-registry-tls" not found Jul 22 00:27:23.398590 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:23.398517 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls podName:e0a1eb7a-b0dc-4804-975f-c601526d229e nodeName:}" failed. No retries permitted until 2026-07-22 00:27:39.398502042 +0000 UTC m=+67.724574239 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls") pod "image-registry-58fcb4cbcd-j5fmf" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e") : secret "image-registry-tls" not found Jul 22 00:27:23.398590 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:23.398530 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls podName:0f69f381-426d-422c-a9df-a007c2e4d0e6 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:39.39852486 +0000 UTC m=+67.724597043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls") pod "dns-default-cdzlx" (UID: "0f69f381-426d-422c-a9df-a007c2e4d0e6") : secret "dns-default-metrics-tls" not found Jul 22 00:27:33.267762 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:33.267732 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g5jvn" Jul 22 00:27:33.418581 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:33.418549 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:33.468560 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:33.468527 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:34.367201 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:34.367146 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:34.467769 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:34.467737 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:34.936089 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:34.936059 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:34.986801 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:34.986751 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:35.244864 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:35.244760 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:35.297926 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:35.297889 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:37.796735 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:37.796684 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:27:37.799587 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:37.799569 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 00:27:37.810976 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:37.810952 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30d68a8f-032a-4d0e-90bf-6f70e5b548f2-original-pull-secret\") pod \"global-pull-secret-syncer-trczs\" (UID: \"30d68a8f-032a-4d0e-90bf-6f70e5b548f2\") " pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:27:37.843860 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:37.843821 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:37.895939 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:37.895909 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:37.897700 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:37.897682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:27:37.900643 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:37.900627 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 00:27:37.908534 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:37.908509 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 00:27:37.908646 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:37.908576 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs podName:d55898f6-dc0e-408a-99cc-9883ae591ac5 nodeName:}" failed. No retries permitted until 2026-07-22 00:28:41.908552345 +0000 UTC m=+130.234624528 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs") pod "network-metrics-daemon-6q2tc" (UID: "d55898f6-dc0e-408a-99cc-9883ae591ac5") : secret "metrics-daemon-secret" not found Jul 22 00:27:37.939242 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:37.939217 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trczs" Jul 22 00:27:37.998370 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:37.998339 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x68d\" (UniqueName: \"kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d\") pod \"network-check-target-qtzz2\" (UID: \"7c8c37c8-072f-4c0b-84a5-e880dcbb7d17\") " pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:27:38.001146 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:38.001127 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 00:27:38.011519 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:38.011498 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 00:27:38.022279 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:38.022228 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x68d\" (UniqueName: \"kubernetes.io/projected/7c8c37c8-072f-4c0b-84a5-e880dcbb7d17-kube-api-access-9x68d\") pod \"network-check-target-qtzz2\" (UID: \"7c8c37c8-072f-4c0b-84a5-e880dcbb7d17\") " pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:27:38.074849 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:38.074744 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-trczs"] Jul 22 00:27:38.077623 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:27:38.077593 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d68a8f_032a_4d0e_90bf_6f70e5b548f2.slice/crio-d5206a0e7a4184d98a13388795f9e6d698f0d7bde06849dc15ab8cb9aa6bacf3 WatchSource:0}: Error finding container d5206a0e7a4184d98a13388795f9e6d698f0d7bde06849dc15ab8cb9aa6bacf3: Status 404 returned error can't find the container with id d5206a0e7a4184d98a13388795f9e6d698f0d7bde06849dc15ab8cb9aa6bacf3 Jul 22 00:27:38.235000 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:38.234972 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rsxjh\"" Jul 22 00:27:38.243009 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:38.242989 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:27:38.357260 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:38.357155 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-trczs" event={"ID":"30d68a8f-032a-4d0e-90bf-6f70e5b548f2","Type":"ContainerStarted","Data":"d5206a0e7a4184d98a13388795f9e6d698f0d7bde06849dc15ab8cb9aa6bacf3"} Jul 22 00:27:38.357705 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:38.357685 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qtzz2"] Jul 22 00:27:38.359925 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:27:38.359900 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8c37c8_072f_4c0b_84a5_e880dcbb7d17.slice/crio-b1c52879a39586b4670f7f205d935ca39e9b784a2a72fa11c937151766f81f34 WatchSource:0}: Error finding container b1c52879a39586b4670f7f205d935ca39e9b784a2a72fa11c937151766f81f34: Status 404 returned error can't find the container with id b1c52879a39586b4670f7f205d935ca39e9b784a2a72fa11c937151766f81f34 Jul 22 00:27:39.309455 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:39.309413 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-zmtpr\" (UID: \"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:27:39.310037 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:39.309550 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 00:27:39.310037 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:39.309627 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert podName:e97ba97d-b165-4bd4-9a46-d8ac502f5d9d nodeName:}" failed. No retries permitted until 2026-07-22 00:28:11.309605719 +0000 UTC m=+99.635677919 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-zmtpr" (UID: "e97ba97d-b165-4bd4-9a46-d8ac502f5d9d") : secret "networking-console-plugin-cert" not found Jul 22 00:27:39.362062 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:39.362020 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qtzz2" event={"ID":"7c8c37c8-072f-4c0b-84a5-e880dcbb7d17","Type":"ContainerStarted","Data":"b1c52879a39586b4670f7f205d935ca39e9b784a2a72fa11c937151766f81f34"} Jul 22 00:27:39.410676 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:39.410631 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:27:39.410881 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:39.410755 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:27:39.410881 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:39.410820 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:39.411001 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:39.410886 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 00:27:39.411001 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:39.410907 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58fcb4cbcd-j5fmf: secret "image-registry-tls" not found Jul 22 00:27:39.411001 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:39.410890 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls podName:0f69f381-426d-422c-a9df-a007c2e4d0e6 nodeName:}" failed. No retries permitted until 2026-07-22 00:28:11.410866757 +0000 UTC m=+99.736938952 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls") pod "dns-default-cdzlx" (UID: "0f69f381-426d-422c-a9df-a007c2e4d0e6") : secret "dns-default-metrics-tls" not found Jul 22 00:27:39.411001 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:27:39.410970 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls podName:e0a1eb7a-b0dc-4804-975f-c601526d229e nodeName:}" failed. No retries permitted until 2026-07-22 00:28:11.410953353 +0000 UTC m=+99.737025538 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls") pod "image-registry-58fcb4cbcd-j5fmf" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e") : secret "image-registry-tls" not found Jul 22 00:27:43.373745 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:43.373706 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qtzz2" event={"ID":"7c8c37c8-072f-4c0b-84a5-e880dcbb7d17","Type":"ContainerStarted","Data":"4da13295b9ac71799e23e26f72c80ddfb6fdcae63fb81538d443bac133f61357"} Jul 22 00:27:43.374207 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:43.373768 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:27:43.375013 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:43.374990 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-trczs" event={"ID":"30d68a8f-032a-4d0e-90bf-6f70e5b548f2","Type":"ContainerStarted","Data":"a6a2844999a2fd02ca2194e8ff0dd15cf3460aef6cbc36af83685107cdd89f91"} Jul 22 00:27:43.389445 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:43.389402 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qtzz2" podStartSLOduration=67.085976278 podStartE2EDuration="1m11.389388437s" totalImagesPullingTime="4.303412159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:43.387836179 +0000 UTC m=+71.713908378" watchObservedRunningTime="2026-07-22 00:27:43.389388437 +0000 UTC m=+71.715460638" Jul 22 00:27:43.400978 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:43.400923 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-trczs" podStartSLOduration=66.811382435 podStartE2EDuration="1m11.40090983s" totalImagesPullingTime="4.589527395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:43.400722931 +0000 UTC m=+71.726795144" watchObservedRunningTime="2026-07-22 00:27:43.40090983 +0000 UTC m=+71.726982085" Jul 22 00:27:56.716732 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:56.716596 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:27:56.769169 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:27:56.769136 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:28:11.337702 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:28:11.337658 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-zmtpr\" (UID: \"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:28:11.338100 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:28:11.337794 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 00:28:11.338100 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:28:11.337861 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert podName:e97ba97d-b165-4bd4-9a46-d8ac502f5d9d nodeName:}" failed. No retries permitted until 2026-07-22 00:29:15.337842119 +0000 UTC m=+163.663914301 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-zmtpr" (UID: "e97ba97d-b165-4bd4-9a46-d8ac502f5d9d") : secret "networking-console-plugin-cert" not found Jul 22 00:28:11.438929 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:28:11.438896 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:28:11.438929 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:28:11.438932 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:28:11.439114 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:28:11.439044 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:28:11.439114 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:28:11.439099 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls podName:0f69f381-426d-422c-a9df-a007c2e4d0e6 nodeName:}" failed. No retries permitted until 2026-07-22 00:29:15.439086524 +0000 UTC m=+163.765158704 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls") pod "dns-default-cdzlx" (UID: "0f69f381-426d-422c-a9df-a007c2e4d0e6") : secret "dns-default-metrics-tls" not found Jul 22 00:28:11.439295 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:28:11.439048 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 00:28:11.439295 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:28:11.439150 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58fcb4cbcd-j5fmf: secret "image-registry-tls" not found Jul 22 00:28:11.439295 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:28:11.439201 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls podName:e0a1eb7a-b0dc-4804-975f-c601526d229e nodeName:}" failed. No retries permitted until 2026-07-22 00:29:15.439188506 +0000 UTC m=+163.765260700 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls") pod "image-registry-58fcb4cbcd-j5fmf" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e") : secret "image-registry-tls" not found Jul 22 00:28:14.380384 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:28:14.380344 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qtzz2" Jul 22 00:28:41.945325 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:28:41.945272 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:28:41.945830 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:28:41.945425 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 00:28:41.945830 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:28:41.945493 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs podName:d55898f6-dc0e-408a-99cc-9883ae591ac5 nodeName:}" failed. No retries permitted until 2026-07-22 00:30:43.945476629 +0000 UTC m=+252.271548826 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs") pod "network-metrics-daemon-6q2tc" (UID: "d55898f6-dc0e-408a-99cc-9883ae591ac5") : secret "metrics-daemon-secret" not found Jul 22 00:29:10.502349 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:29:10.502293 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" podUID="e97ba97d-b165-4bd4-9a46-d8ac502f5d9d" Jul 22 00:29:10.511409 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:29:10.511379 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" podUID="e0a1eb7a-b0dc-4804-975f-c601526d229e" Jul 22 00:29:10.524552 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:29:10.524530 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-cdzlx" podUID="0f69f381-426d-422c-a9df-a007c2e4d0e6" Jul 22 00:29:10.583724 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:29:10.583652 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cdzlx" Jul 22 00:29:10.583850 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:29:10.583652 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:29:10.583850 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:29:10.583663 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:29:12.124721 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:29:12.124683 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:29:15.377856 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:29:15.377791 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-zmtpr\" (UID: \"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:29:15.378311 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:29:15.377935 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 00:29:15.378311 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:29:15.377993 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert podName:e97ba97d-b165-4bd4-9a46-d8ac502f5d9d nodeName:}" failed. No retries permitted until 2026-07-22 00:31:17.377977557 +0000 UTC m=+285.704049737 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-zmtpr" (UID: "e97ba97d-b165-4bd4-9a46-d8ac502f5d9d") : secret "networking-console-plugin-cert" not found Jul 22 00:29:15.478182 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:29:15.478144 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:29:15.478182 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:29:15.478189 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:29:15.478334 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:29:15.478292 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 00:29:15.478334 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:29:15.478317 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58fcb4cbcd-j5fmf: secret "image-registry-tls" not found Jul 22 00:29:15.478334 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:29:15.478299 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:29:15.478433 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:29:15.478379 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls podName:e0a1eb7a-b0dc-4804-975f-c601526d229e nodeName:}" failed. No retries permitted until 2026-07-22 00:31:17.478359959 +0000 UTC m=+285.804432160 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls") pod "image-registry-58fcb4cbcd-j5fmf" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e") : secret "image-registry-tls" not found Jul 22 00:29:15.478433 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:29:15.478400 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls podName:0f69f381-426d-422c-a9df-a007c2e4d0e6 nodeName:}" failed. No retries permitted until 2026-07-22 00:31:17.478391045 +0000 UTC m=+285.804463225 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls") pod "dns-default-cdzlx" (UID: "0f69f381-426d-422c-a9df-a007c2e4d0e6") : secret "dns-default-metrics-tls" not found Jul 22 00:29:25.112226 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:29:25.112138 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:30:05.293557 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:05.293515 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:30:21.332614 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:21.332572 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:30:43.970627 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:43.970576 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:30:43.971206 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:43.970727 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 00:30:43.971206 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:43.970822 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs podName:d55898f6-dc0e-408a-99cc-9883ae591ac5 nodeName:}" failed. No retries permitted until 2026-07-22 00:32:45.970804626 +0000 UTC m=+374.296876810 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs") pod "network-metrics-daemon-6q2tc" (UID: "d55898f6-dc0e-408a-99cc-9883ae591ac5") : secret "metrics-daemon-secret" not found Jul 22 00:30:48.714924 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.714890 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qglr2"] Jul 22 00:30:48.717658 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.717642 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qglr2" Jul 22 00:30:48.720644 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.720625 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 22 00:30:48.721799 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.721758 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 22 00:30:48.721904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.721759 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-cc42q\"" Jul 22 00:30:48.723159 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.723145 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7"] Jul 22 00:30:48.725802 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.725768 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs"] Jul 22 00:30:48.725880 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.725852 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" Jul 22 00:30:48.728033 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.728015 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 22 00:30:48.728670 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.728652 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" Jul 22 00:30:48.728896 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.728876 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 22 00:30:48.729267 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.729250 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 22 00:30:48.729347 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.729274 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 22 00:30:48.729347 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.729280 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nhvfk\"" Jul 22 00:30:48.730445 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.730425 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qglr2"] Jul 22 00:30:48.731151 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.731132 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 22 00:30:48.731151 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.731141 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 22 00:30:48.732236 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.732216 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 22 00:30:48.732331 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.732218 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 22 00:30:48.732331 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.732241 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8d2bx\"" Jul 22 00:30:48.737911 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.737888 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs"] Jul 22 00:30:48.738802 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.738769 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7"] Jul 22 00:30:48.813715 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.813689 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-b522z"] Jul 22 00:30:48.816526 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.816500 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b522z" Jul 22 00:30:48.817504 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.817482 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-sdhl6"] Jul 22 00:30:48.819189 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.819173 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vnnpq\"" Jul 22 00:30:48.820149 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.820133 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w"] Jul 22 00:30:48.820244 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.820232 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:48.822892 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.822874 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j"] Jul 22 00:30:48.823049 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.823017 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" Jul 22 00:30:48.823356 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.823336 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 22 00:30:48.823620 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.823604 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 00:30:48.824204 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.824183 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qf59t\"" Jul 22 00:30:48.824330 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.824192 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 22 00:30:48.824524 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.824509 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 00:30:48.825947 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.825927 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lml8q\"" Jul 22 00:30:48.826034 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.825992 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 22 00:30:48.826132 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.825996 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 22 00:30:48.826132 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.826112 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 22 00:30:48.826321 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.826298 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7584584bbd-c77b7"] Jul 22 00:30:48.826421 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.826406 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:30:48.829257 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.829239 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 22 00:30:48.829520 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.829282 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-s64lm\"" Jul 22 00:30:48.829670 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.829649 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 00:30:48.829911 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.829891 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 00:30:48.830020 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.829990 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 22 00:30:48.830495 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.830158 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-b522z"] Jul 22 00:30:48.830495 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.830243 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:48.831062 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.831035 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 22 00:30:48.832658 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.832642 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w"] Jul 22 00:30:48.834001 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.833641 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 22 00:30:48.834001 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.833913 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-787w4\"" Jul 22 00:30:48.834524 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.834507 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 00:30:48.835066 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.834592 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 22 00:30:48.835765 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.835740 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 22 00:30:48.835884 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.835834 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 00:30:48.836119 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.836102 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 22 00:30:48.837052 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.837031 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-sdhl6"] Jul 22 00:30:48.840093 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.840043 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j"] Jul 22 00:30:48.842552 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.842535 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7584584bbd-c77b7"] Jul 22 00:30:48.905249 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.905221 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37f01742-c77e-4f48-8ab1-e9c4740f3d79-config\") pod \"service-ca-operator-5bcbc94c78-n9fxs\" (UID: \"37f01742-c77e-4f48-8ab1-e9c4740f3d79\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" Jul 22 00:30:48.905373 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.905252 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8dw8\" (UniqueName: \"kubernetes.io/projected/0db5ed04-ffe2-47d9-a85a-1934f3aeb751-kube-api-access-l8dw8\") pod \"volume-data-source-validator-78cf4c6f6-qglr2\" (UID: \"0db5ed04-ffe2-47d9-a85a-1934f3aeb751\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qglr2" Jul 22 00:30:48.905373 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.905273 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp9g9\" (UniqueName: \"kubernetes.io/projected/f0b40554-c78f-463e-bce6-b4ac0b2b1c51-kube-api-access-zp9g9\") pod \"kube-storage-version-migrator-operator-7658d9b749-kfnl7\" (UID: \"f0b40554-c78f-463e-bce6-b4ac0b2b1c51\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" Jul 22 00:30:48.905373 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.905346 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0b40554-c78f-463e-bce6-b4ac0b2b1c51-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-kfnl7\" (UID: \"f0b40554-c78f-463e-bce6-b4ac0b2b1c51\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" Jul 22 00:30:48.905469 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.905379 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37f01742-c77e-4f48-8ab1-e9c4740f3d79-serving-cert\") pod \"service-ca-operator-5bcbc94c78-n9fxs\" (UID: \"37f01742-c77e-4f48-8ab1-e9c4740f3d79\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" Jul 22 00:30:48.905469 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.905399 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0b40554-c78f-463e-bce6-b4ac0b2b1c51-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-kfnl7\" (UID: \"f0b40554-c78f-463e-bce6-b4ac0b2b1c51\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" Jul 22 00:30:48.905469 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.905413 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsdkp\" (UniqueName: \"kubernetes.io/projected/37f01742-c77e-4f48-8ab1-e9c4740f3d79-kube-api-access-nsdkp\") pod \"service-ca-operator-5bcbc94c78-n9fxs\" (UID: \"37f01742-c77e-4f48-8ab1-e9c4740f3d79\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" Jul 22 00:30:48.946099 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:48.946060 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:30:49.005756 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.005694 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qm55w\" (UID: \"353eb386-a09f-42e0-af32-c674bef8994a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" Jul 22 00:30:49.005756 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.005721 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-tbc4j\" (UID: \"a7accd6c-8f60-4679-9ae1-331f59788c1b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:30:49.005756 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.005743 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0b40554-c78f-463e-bce6-b4ac0b2b1c51-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-kfnl7\" (UID: \"f0b40554-c78f-463e-bce6-b4ac0b2b1c51\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" Jul 22 00:30:49.005934 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.005760 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zcbj\" (UniqueName: \"kubernetes.io/projected/353eb386-a09f-42e0-af32-c674bef8994a-kube-api-access-2zcbj\") pod \"cluster-samples-operator-79b7c55f87-qm55w\" (UID: \"353eb386-a09f-42e0-af32-c674bef8994a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" Jul 22 00:30:49.005934 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.005797 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb290190-a2f9-4fd6-8267-b395887f767b-tmp\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.005934 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.005820 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:49.005934 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.005839 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:49.005934 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.005855 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5d6\" (UniqueName: \"kubernetes.io/projected/eb290190-a2f9-4fd6-8267-b395887f767b-kube-api-access-9j5d6\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.005934 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.005874 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-stats-auth\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:49.006108 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.005951 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb290190-a2f9-4fd6-8267-b395887f767b-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.006108 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.005987 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-default-certificate\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:49.006108 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.006010 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a7accd6c-8f60-4679-9ae1-331f59788c1b-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-tbc4j\" (UID: \"a7accd6c-8f60-4679-9ae1-331f59788c1b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:30:49.006108 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.006042 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37f01742-c77e-4f48-8ab1-e9c4740f3d79-serving-cert\") pod \"service-ca-operator-5bcbc94c78-n9fxs\" (UID: \"37f01742-c77e-4f48-8ab1-e9c4740f3d79\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" Jul 22 00:30:49.006108 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.006081 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqbjj\" (UniqueName: \"kubernetes.io/projected/38cd3c0c-5f06-4bf6-b3e7-af8e0669f085-kube-api-access-zqbjj\") pod \"network-check-source-7bc5b6bf85-b522z\" (UID: \"38cd3c0c-5f06-4bf6-b3e7-af8e0669f085\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b522z" Jul 22 00:30:49.006258 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.006141 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n29q\" (UniqueName: \"kubernetes.io/projected/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-kube-api-access-2n29q\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:49.006258 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.006180 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0b40554-c78f-463e-bce6-b4ac0b2b1c51-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-kfnl7\" (UID: \"f0b40554-c78f-463e-bce6-b4ac0b2b1c51\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" Jul 22 00:30:49.006258 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.006204 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsdkp\" (UniqueName: \"kubernetes.io/projected/37f01742-c77e-4f48-8ab1-e9c4740f3d79-kube-api-access-nsdkp\") pod \"service-ca-operator-5bcbc94c78-n9fxs\" (UID: \"37f01742-c77e-4f48-8ab1-e9c4740f3d79\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" Jul 22 00:30:49.006258 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.006234 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eb290190-a2f9-4fd6-8267-b395887f767b-snapshots\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.006442 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.006259 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37f01742-c77e-4f48-8ab1-e9c4740f3d79-config\") pod \"service-ca-operator-5bcbc94c78-n9fxs\" (UID: \"37f01742-c77e-4f48-8ab1-e9c4740f3d79\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" Jul 22 00:30:49.006442 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.006283 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8dw8\" (UniqueName: \"kubernetes.io/projected/0db5ed04-ffe2-47d9-a85a-1934f3aeb751-kube-api-access-l8dw8\") pod \"volume-data-source-validator-78cf4c6f6-qglr2\" (UID: \"0db5ed04-ffe2-47d9-a85a-1934f3aeb751\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qglr2" Jul 22 00:30:49.006442 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.006308 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb290190-a2f9-4fd6-8267-b395887f767b-serving-cert\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.006442 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.006318 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0b40554-c78f-463e-bce6-b4ac0b2b1c51-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-kfnl7\" (UID: \"f0b40554-c78f-463e-bce6-b4ac0b2b1c51\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" Jul 22 00:30:49.006442 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.006334 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp9g9\" (UniqueName: \"kubernetes.io/projected/f0b40554-c78f-463e-bce6-b4ac0b2b1c51-kube-api-access-zp9g9\") pod \"kube-storage-version-migrator-operator-7658d9b749-kfnl7\" (UID: \"f0b40554-c78f-463e-bce6-b4ac0b2b1c51\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" Jul 22 00:30:49.006442 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.006363 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpt9t\" (UniqueName: \"kubernetes.io/projected/a7accd6c-8f60-4679-9ae1-331f59788c1b-kube-api-access-dpt9t\") pod \"cluster-monitoring-operator-76cf8966f8-tbc4j\" (UID: \"a7accd6c-8f60-4679-9ae1-331f59788c1b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:30:49.006442 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.006388 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb290190-a2f9-4fd6-8267-b395887f767b-service-ca-bundle\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.007416 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.007395 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37f01742-c77e-4f48-8ab1-e9c4740f3d79-config\") pod \"service-ca-operator-5bcbc94c78-n9fxs\" (UID: \"37f01742-c77e-4f48-8ab1-e9c4740f3d79\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" Jul 22 00:30:49.008310 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.008291 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37f01742-c77e-4f48-8ab1-e9c4740f3d79-serving-cert\") pod \"service-ca-operator-5bcbc94c78-n9fxs\" (UID: \"37f01742-c77e-4f48-8ab1-e9c4740f3d79\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" Jul 22 00:30:49.008412 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.008396 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0b40554-c78f-463e-bce6-b4ac0b2b1c51-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-kfnl7\" (UID: \"f0b40554-c78f-463e-bce6-b4ac0b2b1c51\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" Jul 22 00:30:49.014336 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.014311 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp9g9\" (UniqueName: \"kubernetes.io/projected/f0b40554-c78f-463e-bce6-b4ac0b2b1c51-kube-api-access-zp9g9\") pod \"kube-storage-version-migrator-operator-7658d9b749-kfnl7\" (UID: \"f0b40554-c78f-463e-bce6-b4ac0b2b1c51\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" Jul 22 00:30:49.014614 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.014597 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsdkp\" (UniqueName: \"kubernetes.io/projected/37f01742-c77e-4f48-8ab1-e9c4740f3d79-kube-api-access-nsdkp\") pod \"service-ca-operator-5bcbc94c78-n9fxs\" (UID: \"37f01742-c77e-4f48-8ab1-e9c4740f3d79\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" Jul 22 00:30:49.014723 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.014707 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8dw8\" (UniqueName: \"kubernetes.io/projected/0db5ed04-ffe2-47d9-a85a-1934f3aeb751-kube-api-access-l8dw8\") pod \"volume-data-source-validator-78cf4c6f6-qglr2\" (UID: \"0db5ed04-ffe2-47d9-a85a-1934f3aeb751\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qglr2" Jul 22 00:30:49.027624 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.027608 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qglr2" Jul 22 00:30:49.037087 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.037071 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" Jul 22 00:30:49.042745 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.042727 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" Jul 22 00:30:49.047839 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.047773 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:30:49.107767 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.107730 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eb290190-a2f9-4fd6-8267-b395887f767b-snapshots\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.107915 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.107807 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb290190-a2f9-4fd6-8267-b395887f767b-serving-cert\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.107915 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.107843 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpt9t\" (UniqueName: \"kubernetes.io/projected/a7accd6c-8f60-4679-9ae1-331f59788c1b-kube-api-access-dpt9t\") pod \"cluster-monitoring-operator-76cf8966f8-tbc4j\" (UID: \"a7accd6c-8f60-4679-9ae1-331f59788c1b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:30:49.107915 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.107872 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb290190-a2f9-4fd6-8267-b395887f767b-service-ca-bundle\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.108079 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.107944 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qm55w\" (UID: \"353eb386-a09f-42e0-af32-c674bef8994a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" Jul 22 00:30:49.108079 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.107977 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-tbc4j\" (UID: \"a7accd6c-8f60-4679-9ae1-331f59788c1b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:30:49.108079 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.108012 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zcbj\" (UniqueName: \"kubernetes.io/projected/353eb386-a09f-42e0-af32-c674bef8994a-kube-api-access-2zcbj\") pod \"cluster-samples-operator-79b7c55f87-qm55w\" (UID: \"353eb386-a09f-42e0-af32-c674bef8994a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" Jul 22 00:30:49.108079 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.108044 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb290190-a2f9-4fd6-8267-b395887f767b-tmp\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.108079 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.108070 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:49.108333 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.108101 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:49.108333 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.108133 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5d6\" (UniqueName: \"kubernetes.io/projected/eb290190-a2f9-4fd6-8267-b395887f767b-kube-api-access-9j5d6\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.108333 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.108159 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-stats-auth\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:49.108333 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.108188 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb290190-a2f9-4fd6-8267-b395887f767b-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.108333 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.108220 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-default-certificate\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:49.108333 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.108251 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a7accd6c-8f60-4679-9ae1-331f59788c1b-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-tbc4j\" (UID: \"a7accd6c-8f60-4679-9ae1-331f59788c1b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:30:49.108333 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.108288 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqbjj\" (UniqueName: \"kubernetes.io/projected/38cd3c0c-5f06-4bf6-b3e7-af8e0669f085-kube-api-access-zqbjj\") pod \"network-check-source-7bc5b6bf85-b522z\" (UID: \"38cd3c0c-5f06-4bf6-b3e7-af8e0669f085\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b522z" Jul 22 00:30:49.108333 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.108315 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n29q\" (UniqueName: \"kubernetes.io/projected/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-kube-api-access-2n29q\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:49.110752 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:49.108929 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 00:30:49.110752 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:49.109012 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls podName:a7accd6c-8f60-4679-9ae1-331f59788c1b nodeName:}" failed. No retries permitted until 2026-07-22 00:30:49.608991594 +0000 UTC m=+257.935063780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-tbc4j" (UID: "a7accd6c-8f60-4679-9ae1-331f59788c1b") : secret "cluster-monitoring-operator-tls" not found Jul 22 00:30:49.110752 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.108931 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb290190-a2f9-4fd6-8267-b395887f767b-tmp\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.110752 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:49.109043 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle podName:09532874-5a40-44cd-b95b-1b3e9ad8ecfd nodeName:}" failed. No retries permitted until 2026-07-22 00:30:49.609025673 +0000 UTC m=+257.935097884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle") pod "router-default-7584584bbd-c77b7" (UID: "09532874-5a40-44cd-b95b-1b3e9ad8ecfd") : configmap references non-existent config key: service-ca.crt Jul 22 00:30:49.110752 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:49.108929 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 00:30:49.110752 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:49.109116 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls podName:353eb386-a09f-42e0-af32-c674bef8994a nodeName:}" failed. No retries permitted until 2026-07-22 00:30:49.609098297 +0000 UTC m=+257.935170476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-qm55w" (UID: "353eb386-a09f-42e0-af32-c674bef8994a") : secret "samples-operator-tls" not found Jul 22 00:30:49.110752 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.109739 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb290190-a2f9-4fd6-8267-b395887f767b-service-ca-bundle\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.110752 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.109922 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a7accd6c-8f60-4679-9ae1-331f59788c1b-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-tbc4j\" (UID: \"a7accd6c-8f60-4679-9ae1-331f59788c1b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:30:49.110752 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:49.110085 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 00:30:49.110752 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.110116 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eb290190-a2f9-4fd6-8267-b395887f767b-snapshots\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.110752 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:49.110135 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs podName:09532874-5a40-44cd-b95b-1b3e9ad8ecfd nodeName:}" failed. No retries permitted until 2026-07-22 00:30:49.610119161 +0000 UTC m=+257.936191343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs") pod "router-default-7584584bbd-c77b7" (UID: "09532874-5a40-44cd-b95b-1b3e9ad8ecfd") : secret "router-metrics-certs-default" not found Jul 22 00:30:49.112056 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.112021 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb290190-a2f9-4fd6-8267-b395887f767b-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.112592 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.112567 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-stats-auth\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:49.113069 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.113043 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-default-certificate\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:49.113712 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.113678 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb290190-a2f9-4fd6-8267-b395887f767b-serving-cert\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.119020 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.118992 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5d6\" (UniqueName: \"kubernetes.io/projected/eb290190-a2f9-4fd6-8267-b395887f767b-kube-api-access-9j5d6\") pod \"insights-operator-555d9d7885-sdhl6\" (UID: \"eb290190-a2f9-4fd6-8267-b395887f767b\") " pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.120147 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.120125 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zcbj\" (UniqueName: \"kubernetes.io/projected/353eb386-a09f-42e0-af32-c674bef8994a-kube-api-access-2zcbj\") pod \"cluster-samples-operator-79b7c55f87-qm55w\" (UID: \"353eb386-a09f-42e0-af32-c674bef8994a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" Jul 22 00:30:49.120443 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.120425 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n29q\" (UniqueName: \"kubernetes.io/projected/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-kube-api-access-2n29q\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:49.120895 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.120877 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpt9t\" (UniqueName: \"kubernetes.io/projected/a7accd6c-8f60-4679-9ae1-331f59788c1b-kube-api-access-dpt9t\") pod \"cluster-monitoring-operator-76cf8966f8-tbc4j\" (UID: \"a7accd6c-8f60-4679-9ae1-331f59788c1b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:30:49.121797 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.121684 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqbjj\" (UniqueName: \"kubernetes.io/projected/38cd3c0c-5f06-4bf6-b3e7-af8e0669f085-kube-api-access-zqbjj\") pod \"network-check-source-7bc5b6bf85-b522z\" (UID: \"38cd3c0c-5f06-4bf6-b3e7-af8e0669f085\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b522z" Jul 22 00:30:49.128259 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.128184 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b522z" Jul 22 00:30:49.137680 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.137560 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-sdhl6" Jul 22 00:30:49.182435 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.182404 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qglr2"] Jul 22 00:30:49.185575 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:30:49.185543 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db5ed04_ffe2_47d9_a85a_1934f3aeb751.slice/crio-c0cf9da911ca1e0e61bd26cde123fcdee57365bd6313c7d2cb621591fb165115 WatchSource:0}: Error finding container c0cf9da911ca1e0e61bd26cde123fcdee57365bd6313c7d2cb621591fb165115: Status 404 returned error can't find the container with id c0cf9da911ca1e0e61bd26cde123fcdee57365bd6313c7d2cb621591fb165115 Jul 22 00:30:49.271820 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.271723 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-b522z"] Jul 22 00:30:49.274040 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:30:49.274014 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38cd3c0c_5f06_4bf6_b3e7_af8e0669f085.slice/crio-0023716c981cc35e18d9db11b12837f5413603e7e6a88d7c9c24773c9caa0347 WatchSource:0}: Error finding container 0023716c981cc35e18d9db11b12837f5413603e7e6a88d7c9c24773c9caa0347: Status 404 returned error can't find the container with id 0023716c981cc35e18d9db11b12837f5413603e7e6a88d7c9c24773c9caa0347 Jul 22 00:30:49.284932 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.284909 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-sdhl6"] Jul 22 00:30:49.289076 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:30:49.289054 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb290190_a2f9_4fd6_8267_b395887f767b.slice/crio-5d84ceb1573a8ccb562826cb5484b527bf111702caba448bdf0a9ece9840d315 WatchSource:0}: Error finding container 5d84ceb1573a8ccb562826cb5484b527bf111702caba448bdf0a9ece9840d315: Status 404 returned error can't find the container with id 5d84ceb1573a8ccb562826cb5484b527bf111702caba448bdf0a9ece9840d315 Jul 22 00:30:49.398548 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.398528 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs"] Jul 22 00:30:49.399755 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.399725 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7"] Jul 22 00:30:49.400048 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:30:49.400020 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b40554_c78f_463e_bce6_b4ac0b2b1c51.slice/crio-5af28c6451e9dade8b93de907dc3b14bc47edc78e375a07fb4f12581515ec4ae WatchSource:0}: Error finding container 5af28c6451e9dade8b93de907dc3b14bc47edc78e375a07fb4f12581515ec4ae: Status 404 returned error can't find the container with id 5af28c6451e9dade8b93de907dc3b14bc47edc78e375a07fb4f12581515ec4ae Jul 22 00:30:49.400833 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:30:49.400727 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f01742_c77e_4f48_8ab1_e9c4740f3d79.slice/crio-11fb28288e905b325f13699c384edcc8a5e8853cbbe1e52973f7b5af0a365cb3 WatchSource:0}: Error finding container 11fb28288e905b325f13699c384edcc8a5e8853cbbe1e52973f7b5af0a365cb3: Status 404 returned error can't find the container with id 11fb28288e905b325f13699c384edcc8a5e8853cbbe1e52973f7b5af0a365cb3 Jul 22 00:30:49.612662 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.612604 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qm55w\" (UID: \"353eb386-a09f-42e0-af32-c674bef8994a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" Jul 22 00:30:49.612662 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.612638 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-tbc4j\" (UID: \"a7accd6c-8f60-4679-9ae1-331f59788c1b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:30:49.612815 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.612666 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:49.612815 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.612694 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:49.612815 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:49.612760 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 00:30:49.612815 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:49.612765 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 00:30:49.612995 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:49.612831 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls podName:353eb386-a09f-42e0-af32-c674bef8994a nodeName:}" failed. No retries permitted until 2026-07-22 00:30:50.612814409 +0000 UTC m=+258.938886605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-qm55w" (UID: "353eb386-a09f-42e0-af32-c674bef8994a") : secret "samples-operator-tls" not found Jul 22 00:30:49.612995 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:49.612854 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle podName:09532874-5a40-44cd-b95b-1b3e9ad8ecfd nodeName:}" failed. No retries permitted until 2026-07-22 00:30:50.61284411 +0000 UTC m=+258.938916294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle") pod "router-default-7584584bbd-c77b7" (UID: "09532874-5a40-44cd-b95b-1b3e9ad8ecfd") : configmap references non-existent config key: service-ca.crt Jul 22 00:30:49.612995 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:49.612859 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 00:30:49.612995 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:49.612879 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs podName:09532874-5a40-44cd-b95b-1b3e9ad8ecfd nodeName:}" failed. No retries permitted until 2026-07-22 00:30:50.612864562 +0000 UTC m=+258.938936747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs") pod "router-default-7584584bbd-c77b7" (UID: "09532874-5a40-44cd-b95b-1b3e9ad8ecfd") : secret "router-metrics-certs-default" not found Jul 22 00:30:49.612995 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:49.612920 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls podName:a7accd6c-8f60-4679-9ae1-331f59788c1b nodeName:}" failed. No retries permitted until 2026-07-22 00:30:50.612903877 +0000 UTC m=+258.938976069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-tbc4j" (UID: "a7accd6c-8f60-4679-9ae1-331f59788c1b") : secret "cluster-monitoring-operator-tls" not found Jul 22 00:30:49.828354 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.828316 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-sdhl6" event={"ID":"eb290190-a2f9-4fd6-8267-b395887f767b","Type":"ContainerStarted","Data":"5d84ceb1573a8ccb562826cb5484b527bf111702caba448bdf0a9ece9840d315"} Jul 22 00:30:49.830048 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.830016 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b522z" event={"ID":"38cd3c0c-5f06-4bf6-b3e7-af8e0669f085","Type":"ContainerStarted","Data":"d1e00a021e6f6de937bafb707b3dc3e615b8c5ba9cf77537093fa385b5be69f8"} Jul 22 00:30:49.830174 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.830057 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b522z" event={"ID":"38cd3c0c-5f06-4bf6-b3e7-af8e0669f085","Type":"ContainerStarted","Data":"0023716c981cc35e18d9db11b12837f5413603e7e6a88d7c9c24773c9caa0347"} Jul 22 00:30:49.831297 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.831271 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" event={"ID":"37f01742-c77e-4f48-8ab1-e9c4740f3d79","Type":"ContainerStarted","Data":"11fb28288e905b325f13699c384edcc8a5e8853cbbe1e52973f7b5af0a365cb3"} Jul 22 00:30:49.832495 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.832463 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qglr2" event={"ID":"0db5ed04-ffe2-47d9-a85a-1934f3aeb751","Type":"ContainerStarted","Data":"c0cf9da911ca1e0e61bd26cde123fcdee57365bd6313c7d2cb621591fb165115"} Jul 22 00:30:49.833738 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.833710 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" event={"ID":"f0b40554-c78f-463e-bce6-b4ac0b2b1c51","Type":"ContainerStarted","Data":"5af28c6451e9dade8b93de907dc3b14bc47edc78e375a07fb4f12581515ec4ae"} Jul 22 00:30:49.845603 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:49.845563 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b522z" podStartSLOduration=1.845550437 podStartE2EDuration="1.845550437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:30:49.845012663 +0000 UTC m=+258.171084865" watchObservedRunningTime="2026-07-22 00:30:49.845550437 +0000 UTC m=+258.171622639" Jul 22 00:30:50.621187 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:50.620567 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qm55w\" (UID: \"353eb386-a09f-42e0-af32-c674bef8994a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" Jul 22 00:30:50.621187 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:50.620621 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-tbc4j\" (UID: \"a7accd6c-8f60-4679-9ae1-331f59788c1b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:30:50.621187 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:50.620665 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:50.621187 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:50.620696 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:50.621187 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:50.620705 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 00:30:50.621187 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:50.620846 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls podName:353eb386-a09f-42e0-af32-c674bef8994a nodeName:}" failed. No retries permitted until 2026-07-22 00:30:52.620799146 +0000 UTC m=+260.946871340 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-qm55w" (UID: "353eb386-a09f-42e0-af32-c674bef8994a") : secret "samples-operator-tls" not found Jul 22 00:30:50.621187 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:50.620919 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle podName:09532874-5a40-44cd-b95b-1b3e9ad8ecfd nodeName:}" failed. No retries permitted until 2026-07-22 00:30:52.620902982 +0000 UTC m=+260.946975165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle") pod "router-default-7584584bbd-c77b7" (UID: "09532874-5a40-44cd-b95b-1b3e9ad8ecfd") : configmap references non-existent config key: service-ca.crt Jul 22 00:30:50.621187 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:50.620995 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 00:30:50.621187 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:50.621034 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls podName:a7accd6c-8f60-4679-9ae1-331f59788c1b nodeName:}" failed. No retries permitted until 2026-07-22 00:30:52.621023868 +0000 UTC m=+260.947096053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-tbc4j" (UID: "a7accd6c-8f60-4679-9ae1-331f59788c1b") : secret "cluster-monitoring-operator-tls" not found Jul 22 00:30:50.621187 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:50.621091 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 00:30:50.621187 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:50.621120 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs podName:09532874-5a40-44cd-b95b-1b3e9ad8ecfd nodeName:}" failed. No retries permitted until 2026-07-22 00:30:52.621109656 +0000 UTC m=+260.947181839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs") pod "router-default-7584584bbd-c77b7" (UID: "09532874-5a40-44cd-b95b-1b3e9ad8ecfd") : secret "router-metrics-certs-default" not found Jul 22 00:30:51.842142 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:51.842107 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qglr2" event={"ID":"0db5ed04-ffe2-47d9-a85a-1934f3aeb751","Type":"ContainerStarted","Data":"084340c8bb20f7a22a0c7b406f824ff4cf8b0589447cf7084cb902a472f63a7a"} Jul 22 00:30:51.859472 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:51.859417 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qglr2" podStartSLOduration=2.245446172 podStartE2EDuration="3.859400786s" totalImagesPullingTime="1.613954614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:30:51.858340566 +0000 UTC m=+260.184412769" watchObservedRunningTime="2026-07-22 00:30:51.859400786 +0000 UTC m=+260.185472987" Jul 22 00:30:51.932197 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:51.932160 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:30:52.639528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:52.639490 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qm55w\" (UID: \"353eb386-a09f-42e0-af32-c674bef8994a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" Jul 22 00:30:52.639528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:52.639534 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-tbc4j\" (UID: \"a7accd6c-8f60-4679-9ae1-331f59788c1b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:30:52.639772 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:52.639571 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:52.639772 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:52.639600 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:52.639772 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:52.639650 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 00:30:52.639772 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:52.639712 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls podName:353eb386-a09f-42e0-af32-c674bef8994a nodeName:}" failed. No retries permitted until 2026-07-22 00:30:56.639694578 +0000 UTC m=+264.965766773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-qm55w" (UID: "353eb386-a09f-42e0-af32-c674bef8994a") : secret "samples-operator-tls" not found Jul 22 00:30:52.639772 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:52.639722 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 00:30:52.639772 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:52.639743 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle podName:09532874-5a40-44cd-b95b-1b3e9ad8ecfd nodeName:}" failed. No retries permitted until 2026-07-22 00:30:56.639733722 +0000 UTC m=+264.965805904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle") pod "router-default-7584584bbd-c77b7" (UID: "09532874-5a40-44cd-b95b-1b3e9ad8ecfd") : configmap references non-existent config key: service-ca.crt Jul 22 00:30:52.639772 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:52.639722 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 00:30:52.640094 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:52.639805 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls podName:a7accd6c-8f60-4679-9ae1-331f59788c1b nodeName:}" failed. No retries permitted until 2026-07-22 00:30:56.639772007 +0000 UTC m=+264.965844187 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-tbc4j" (UID: "a7accd6c-8f60-4679-9ae1-331f59788c1b") : secret "cluster-monitoring-operator-tls" not found Jul 22 00:30:52.640094 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:52.639828 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs podName:09532874-5a40-44cd-b95b-1b3e9ad8ecfd nodeName:}" failed. No retries permitted until 2026-07-22 00:30:56.63981723 +0000 UTC m=+264.965889414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs") pod "router-default-7584584bbd-c77b7" (UID: "09532874-5a40-44cd-b95b-1b3e9ad8ecfd") : secret "router-metrics-certs-default" not found Jul 22 00:30:52.848103 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:52.848062 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-sdhl6" event={"ID":"eb290190-a2f9-4fd6-8267-b395887f767b","Type":"ContainerStarted","Data":"6a54d18f04fe291c65552c4dec4514f6f5822176c216534a663e2d7b9c8d4d41"} Jul 22 00:30:52.849637 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:52.849591 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" event={"ID":"37f01742-c77e-4f48-8ab1-e9c4740f3d79","Type":"ContainerStarted","Data":"79a72fa55a8be87c5fee53814592474dd4c9cb444d5a42a2581935da6904d864"} Jul 22 00:30:52.851852 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:52.851769 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" event={"ID":"f0b40554-c78f-463e-bce6-b4ac0b2b1c51","Type":"ContainerStarted","Data":"fdb15bcd48ccdd6fb1160819072ff5c7772828c75a733d38efb67bb867c18e01"} Jul 22 00:30:52.866570 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:52.866075 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-sdhl6" podStartSLOduration=1.408053375 podStartE2EDuration="4.866056746s" totalImagesPullingTime="3.458003371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:30:52.865546764 +0000 UTC m=+261.191618969" watchObservedRunningTime="2026-07-22 00:30:52.866056746 +0000 UTC m=+261.192128949" Jul 22 00:30:52.885813 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:52.884983 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" podStartSLOduration=1.543794744 podStartE2EDuration="4.884965377s" totalImagesPullingTime="3.341170633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:30:52.884207525 +0000 UTC m=+261.210279727" watchObservedRunningTime="2026-07-22 00:30:52.884965377 +0000 UTC m=+261.211037578" Jul 22 00:30:52.899898 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:52.899807 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" podStartSLOduration=1.556079014 podStartE2EDuration="4.899765575s" totalImagesPullingTime="3.343686561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:30:52.898430933 +0000 UTC m=+261.224503136" watchObservedRunningTime="2026-07-22 00:30:52.899765575 +0000 UTC m=+261.225837779" Jul 22 00:30:53.819847 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:53.819810 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:30:53.914819 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:53.914787 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:30:53.976581 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:53.976547 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-96fng"] Jul 22 00:30:53.985798 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:53.985754 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:53.989137 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:53.988933 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 00:30:53.989137 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:53.988984 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 00:30:53.989137 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:53.988941 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qsbbh\"" Jul 22 00:30:53.991720 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:53.991693 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-96fng"] Jul 22 00:30:53.996615 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:53.996592 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:30:54.050502 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.050476 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l962z\" (UniqueName: \"kubernetes.io/projected/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-api-access-l962z\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:54.050636 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.050509 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:54.050636 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.050528 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:54.050636 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.050584 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a087a509-8906-4b67-a39d-bf24cd20c7da-crio-socket\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:54.050802 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.050689 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a087a509-8906-4b67-a39d-bf24cd20c7da-data-volume\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:54.094774 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.094718 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:30:54.151213 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.151191 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l962z\" (UniqueName: \"kubernetes.io/projected/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-api-access-l962z\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:54.151299 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.151220 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:54.151299 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.151239 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:54.151421 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:54.151311 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 00:30:54.151421 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:54.151370 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:30:54.151421 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:54.151375 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm podName:a087a509-8906-4b67-a39d-bf24cd20c7da nodeName:}" failed. No retries permitted until 2026-07-22 00:30:54.651357966 +0000 UTC m=+262.977430151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-96fng" (UID: "a087a509-8906-4b67-a39d-bf24cd20c7da") : configmap "kube-rbac-proxy" not found Jul 22 00:30:54.151549 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.151463 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a087a509-8906-4b67-a39d-bf24cd20c7da-crio-socket\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:54.151549 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:54.151494 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls podName:a087a509-8906-4b67-a39d-bf24cd20c7da nodeName:}" failed. No retries permitted until 2026-07-22 00:30:54.651478739 +0000 UTC m=+262.977550936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-96fng" (UID: "a087a509-8906-4b67-a39d-bf24cd20c7da") : secret "insights-runtime-extractor-tls" not found Jul 22 00:30:54.151549 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.151519 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a087a509-8906-4b67-a39d-bf24cd20c7da-crio-socket\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:54.151692 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.151572 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a087a509-8906-4b67-a39d-bf24cd20c7da-data-volume\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:54.151869 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.151853 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a087a509-8906-4b67-a39d-bf24cd20c7da-data-volume\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:54.159880 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.159862 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l962z\" (UniqueName: \"kubernetes.io/projected/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-api-access-l962z\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:54.655505 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.655473 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:54.655680 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:54.655511 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:54.655680 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:54.655608 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 00:30:54.655680 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:54.655659 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:30:54.655803 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:54.655681 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm podName:a087a509-8906-4b67-a39d-bf24cd20c7da nodeName:}" failed. No retries permitted until 2026-07-22 00:30:55.655661979 +0000 UTC m=+263.981734158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-96fng" (UID: "a087a509-8906-4b67-a39d-bf24cd20c7da") : configmap "kube-rbac-proxy" not found Jul 22 00:30:54.655803 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:54.655708 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls podName:a087a509-8906-4b67-a39d-bf24cd20c7da nodeName:}" failed. No retries permitted until 2026-07-22 00:30:55.655698127 +0000 UTC m=+263.981770318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-96fng" (UID: "a087a509-8906-4b67-a39d-bf24cd20c7da") : secret "insights-runtime-extractor-tls" not found Jul 22 00:30:55.104007 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:55.103974 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:30:55.206789 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:55.206755 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:30:55.662881 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:55.662846 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:55.663055 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:55.662899 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:55.663055 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:55.662964 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 00:30:55.663055 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:55.663029 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm podName:a087a509-8906-4b67-a39d-bf24cd20c7da nodeName:}" failed. No retries permitted until 2026-07-22 00:30:57.663013112 +0000 UTC m=+265.989085292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-96fng" (UID: "a087a509-8906-4b67-a39d-bf24cd20c7da") : configmap "kube-rbac-proxy" not found Jul 22 00:30:55.663055 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:55.663046 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:30:55.663226 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:55.663144 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls podName:a087a509-8906-4b67-a39d-bf24cd20c7da nodeName:}" failed. No retries permitted until 2026-07-22 00:30:57.663126557 +0000 UTC m=+265.989198737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-96fng" (UID: "a087a509-8906-4b67-a39d-bf24cd20c7da") : secret "insights-runtime-extractor-tls" not found Jul 22 00:30:56.671393 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:56.671352 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qm55w\" (UID: \"353eb386-a09f-42e0-af32-c674bef8994a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" Jul 22 00:30:56.671393 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:56.671393 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-tbc4j\" (UID: \"a7accd6c-8f60-4679-9ae1-331f59788c1b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:30:56.671976 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:56.671505 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 00:30:56.671976 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:56.671550 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:56.671976 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:56.671571 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls podName:353eb386-a09f-42e0-af32-c674bef8994a nodeName:}" failed. No retries permitted until 2026-07-22 00:31:04.671555121 +0000 UTC m=+272.997627302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-qm55w" (UID: "353eb386-a09f-42e0-af32-c674bef8994a") : secret "samples-operator-tls" not found Jul 22 00:30:56.671976 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:56.671511 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 00:30:56.671976 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:56.671613 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls podName:a7accd6c-8f60-4679-9ae1-331f59788c1b nodeName:}" failed. No retries permitted until 2026-07-22 00:31:04.671606933 +0000 UTC m=+272.997679114 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-tbc4j" (UID: "a7accd6c-8f60-4679-9ae1-331f59788c1b") : secret "cluster-monitoring-operator-tls" not found Jul 22 00:30:56.671976 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:56.671624 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 00:30:56.671976 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:56.671675 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs podName:09532874-5a40-44cd-b95b-1b3e9ad8ecfd nodeName:}" failed. No retries permitted until 2026-07-22 00:31:04.671661075 +0000 UTC m=+272.997733256 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs") pod "router-default-7584584bbd-c77b7" (UID: "09532874-5a40-44cd-b95b-1b3e9ad8ecfd") : secret "router-metrics-certs-default" not found Jul 22 00:30:56.671976 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:56.671695 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:30:56.671976 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:56.671855 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle podName:09532874-5a40-44cd-b95b-1b3e9ad8ecfd nodeName:}" failed. No retries permitted until 2026-07-22 00:31:04.671842286 +0000 UTC m=+272.997914469 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle") pod "router-default-7584584bbd-c77b7" (UID: "09532874-5a40-44cd-b95b-1b3e9ad8ecfd") : configmap references non-existent config key: service-ca.crt Jul 22 00:30:56.733867 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:56.733835 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:30:56.784666 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:56.784645 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kxz6k_c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5/dns-node-resolver/0.log" Jul 22 00:30:56.834324 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:56.834293 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:30:57.593415 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:57.593376 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:30:57.680729 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:57.680689 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:57.680729 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:57.680729 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:30:57.681141 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:57.680835 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 00:30:57.681141 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:57.680885 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:30:57.681141 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:57.680898 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm podName:a087a509-8906-4b67-a39d-bf24cd20c7da nodeName:}" failed. No retries permitted until 2026-07-22 00:31:01.680883105 +0000 UTC m=+270.006955285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-96fng" (UID: "a087a509-8906-4b67-a39d-bf24cd20c7da") : configmap "kube-rbac-proxy" not found Jul 22 00:30:57.681141 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:30:57.680929 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls podName:a087a509-8906-4b67-a39d-bf24cd20c7da nodeName:}" failed. No retries permitted until 2026-07-22 00:31:01.680916981 +0000 UTC m=+270.006989160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-96fng" (UID: "a087a509-8906-4b67-a39d-bf24cd20c7da") : secret "insights-runtime-extractor-tls" not found Jul 22 00:30:57.786077 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:57.786052 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lv725_d60d84ba-a0ae-4e46-9fdf-7830c7d5e798/node-ca/0.log" Jul 22 00:30:58.385587 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:58.385530 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5kzgh_2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a/serve-healthcheck-canary/0.log" Jul 22 00:30:59.387537 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:30:59.387506 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-kfnl7_f0b40554-c78f-463e-bce6-b4ac0b2b1c51/kube-storage-version-migrator-operator/0.log" Jul 22 00:31:00.334092 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:00.334056 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:01.087377 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:01.087334 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:01.217155 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:01.217123 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:01.385935 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:01.385839 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:01.486891 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:01.486857 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:01.587271 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:01.587240 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:01.711352 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:01.711266 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:31:01.711352 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:01.711304 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:31:01.711543 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:01.711386 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 00:31:01.711543 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:01.711467 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm podName:a087a509-8906-4b67-a39d-bf24cd20c7da nodeName:}" failed. No retries permitted until 2026-07-22 00:31:09.711449858 +0000 UTC m=+278.037522038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-96fng" (UID: "a087a509-8906-4b67-a39d-bf24cd20c7da") : configmap "kube-rbac-proxy" not found Jul 22 00:31:01.711543 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:01.711489 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:31:01.711688 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:01.711583 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls podName:a087a509-8906-4b67-a39d-bf24cd20c7da nodeName:}" failed. No retries permitted until 2026-07-22 00:31:09.711563745 +0000 UTC m=+278.037635947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-96fng" (UID: "a087a509-8906-4b67-a39d-bf24cd20c7da") : secret "insights-runtime-extractor-tls" not found Jul 22 00:31:02.172648 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:02.172615 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:02.817955 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:02.817924 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:03.190395 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:03.190298 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:03.386367 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:03.386330 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:03.593185 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:03.593144 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:03.695663 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:03.695623 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:03.990913 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:03.990844 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:04.585307 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:04.585264 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:04.733384 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:04.733356 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qm55w\" (UID: \"353eb386-a09f-42e0-af32-c674bef8994a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" Jul 22 00:31:04.733530 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:04.733388 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-tbc4j\" (UID: \"a7accd6c-8f60-4679-9ae1-331f59788c1b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:31:04.733530 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:04.733433 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:31:04.733530 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:04.733519 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 00:31:04.733647 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:04.733533 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 00:31:04.733647 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:04.733549 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:31:04.733647 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:04.733578 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs podName:09532874-5a40-44cd-b95b-1b3e9ad8ecfd nodeName:}" failed. No retries permitted until 2026-07-22 00:31:20.733560776 +0000 UTC m=+289.059632978 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs") pod "router-default-7584584bbd-c77b7" (UID: "09532874-5a40-44cd-b95b-1b3e9ad8ecfd") : secret "router-metrics-certs-default" not found Jul 22 00:31:04.733647 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:04.733599 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls podName:a7accd6c-8f60-4679-9ae1-331f59788c1b nodeName:}" failed. No retries permitted until 2026-07-22 00:31:20.733589851 +0000 UTC m=+289.059662030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-tbc4j" (UID: "a7accd6c-8f60-4679-9ae1-331f59788c1b") : secret "cluster-monitoring-operator-tls" not found Jul 22 00:31:04.733647 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:04.733626 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle podName:09532874-5a40-44cd-b95b-1b3e9ad8ecfd nodeName:}" failed. No retries permitted until 2026-07-22 00:31:20.733610688 +0000 UTC m=+289.059682868 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle") pod "router-default-7584584bbd-c77b7" (UID: "09532874-5a40-44cd-b95b-1b3e9ad8ecfd") : configmap references non-existent config key: service-ca.crt Jul 22 00:31:04.735793 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:04.735752 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/353eb386-a09f-42e0-af32-c674bef8994a-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qm55w\" (UID: \"353eb386-a09f-42e0-af32-c674bef8994a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" Jul 22 00:31:04.743639 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:04.743618 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" Jul 22 00:31:04.863235 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:04.863171 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w"] Jul 22 00:31:04.959419 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:04.959396 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:05.010862 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:05.010840 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:05.188642 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:05.188572 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:05.416631 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:05.416600 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:05.586156 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:05.586133 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:05.784890 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:05.784857 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:05.892733 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:05.892671 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" event={"ID":"353eb386-a09f-42e0-af32-c674bef8994a","Type":"ContainerStarted","Data":"681bb1c5551875657954011eae463ef1915f6c91c717b410ff8fc92c7f2db4a6"} Jul 22 00:31:06.788097 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:06.788056 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:06.987068 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:06.987033 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:07.089376 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:07.089270 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:07.579538 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:07.579507 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:07.646054 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:07.646014 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:08.384502 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:08.384471 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:08.588036 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:08.588007 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:08.808496 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:08.808467 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:08.903225 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:08.903193 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" event={"ID":"353eb386-a09f-42e0-af32-c674bef8994a","Type":"ContainerStarted","Data":"156c1bdbc77da5a0bcbe7ca0a56ddba8bb36275912ac7809888790042c708e2f"} Jul 22 00:31:08.903336 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:08.903246 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" event={"ID":"353eb386-a09f-42e0-af32-c674bef8994a","Type":"ContainerStarted","Data":"6da54e5558cf0d6aac1e93b4e49e05d13cfe33afb21ef148b6b0bab5e99c943b"} Jul 22 00:31:08.918949 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:08.918907 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qm55w" podStartSLOduration=17.393899893 podStartE2EDuration="20.918889867s" totalImagesPullingTime="3.524989974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:08.918293648 +0000 UTC m=+277.244365849" watchObservedRunningTime="2026-07-22 00:31:08.918889867 +0000 UTC m=+277.244962070" Jul 22 00:31:09.355528 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:09.355498 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:09.409641 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:09.409611 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:09.776117 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:09.776091 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:31:09.776275 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:09.776124 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:31:09.776275 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:09.776236 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:31:09.776401 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:09.776286 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls podName:a087a509-8906-4b67-a39d-bf24cd20c7da nodeName:}" failed. No retries permitted until 2026-07-22 00:31:25.776273295 +0000 UTC m=+294.102345476 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-96fng" (UID: "a087a509-8906-4b67-a39d-bf24cd20c7da") : secret "insights-runtime-extractor-tls" not found Jul 22 00:31:09.776676 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:09.776652 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a087a509-8906-4b67-a39d-bf24cd20c7da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:31:09.790406 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:09.790374 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:10.187741 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:10.187647 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:10.584159 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:10.584129 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:10.878233 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:10.878160 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:10.978550 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:10.978480 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:11.028042 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:11.028015 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:11.173487 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:11.173403 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:11.389153 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:11.389125 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:11.560265 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:11.560240 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:11.660204 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:11.660176 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:11.790384 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:11.790354 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:11.954736 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:11.954664 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:12.050219 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:12.050184 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:12.355157 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:12.355096 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:12.456999 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:12.456961 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:12.934422 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:12.934384 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:13.084823 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:13.084766 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:13.428980 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:13.428950 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:13.487516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:13.487491 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:13.584866 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:13.584834 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" podUID="e0a1eb7a-b0dc-4804-975f-c601526d229e" Jul 22 00:31:13.584866 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:13.584835 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" podUID="e97ba97d-b165-4bd4-9a46-d8ac502f5d9d" Jul 22 00:31:13.584980 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:13.584835 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-cdzlx" podUID="0f69f381-426d-422c-a9df-a007c2e4d0e6" Jul 22 00:31:13.920821 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:13.920796 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:31:13.920955 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:13.920798 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:31:13.921008 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:13.920799 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cdzlx" Jul 22 00:31:17.433134 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.433089 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-zmtpr\" (UID: \"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:31:17.435625 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.435601 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e97ba97d-b165-4bd4-9a46-d8ac502f5d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-zmtpr\" (UID: \"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:31:17.524329 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.524302 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-45lzm\"" Jul 22 00:31:17.532330 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.532310 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" Jul 22 00:31:17.533618 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.533594 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:31:17.533699 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.533627 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:31:17.535974 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.535946 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f69f381-426d-422c-a9df-a007c2e4d0e6-metrics-tls\") pod \"dns-default-cdzlx\" (UID: \"0f69f381-426d-422c-a9df-a007c2e4d0e6\") " pod="openshift-dns/dns-default-cdzlx" Jul 22 00:31:17.536196 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.536176 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls\") pod \"image-registry-58fcb4cbcd-j5fmf\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:31:17.653591 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.653561 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr"] Jul 22 00:31:17.656236 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:31:17.656214 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97ba97d_b165_4bd4_9a46_d8ac502f5d9d.slice/crio-dce6ba3cc06fe6166c07f4bb3f450a97ada6d00dff062df8f34ac7b65baa6bed WatchSource:0}: Error finding container dce6ba3cc06fe6166c07f4bb3f450a97ada6d00dff062df8f34ac7b65baa6bed: Status 404 returned error can't find the container with id dce6ba3cc06fe6166c07f4bb3f450a97ada6d00dff062df8f34ac7b65baa6bed Jul 22 00:31:17.824024 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.824001 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nfkpn\"" Jul 22 00:31:17.824129 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.824069 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kstbg\"" Jul 22 00:31:17.831522 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.831505 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cdzlx" Jul 22 00:31:17.831607 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.831594 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:31:17.942747 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.942671 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" event={"ID":"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d","Type":"ContainerStarted","Data":"dce6ba3cc06fe6166c07f4bb3f450a97ada6d00dff062df8f34ac7b65baa6bed"} Jul 22 00:31:17.965172 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.965148 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cdzlx"] Jul 22 00:31:17.967225 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:31:17.967182 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f69f381_426d_422c_a9df_a007c2e4d0e6.slice/crio-dd3c45a5901b7b385fd8e15eb539436854fdef3c6ba5b64046e2817b6fec62ef WatchSource:0}: Error finding container dd3c45a5901b7b385fd8e15eb539436854fdef3c6ba5b64046e2817b6fec62ef: Status 404 returned error can't find the container with id dd3c45a5901b7b385fd8e15eb539436854fdef3c6ba5b64046e2817b6fec62ef Jul 22 00:31:17.980558 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:17.980520 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58fcb4cbcd-j5fmf"] Jul 22 00:31:17.982249 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:31:17.982229 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a1eb7a_b0dc_4804_975f_c601526d229e.slice/crio-efb218e33ab871bcfb093d92184eb583b77e08be40f9108acace78225ad31f49 WatchSource:0}: Error finding container efb218e33ab871bcfb093d92184eb583b77e08be40f9108acace78225ad31f49: Status 404 returned error can't find the container with id efb218e33ab871bcfb093d92184eb583b77e08be40f9108acace78225ad31f49 Jul 22 00:31:18.173953 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:18.173876 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:18.224300 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:18.224264 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:18.948162 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:18.948023 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cdzlx" event={"ID":"0f69f381-426d-422c-a9df-a007c2e4d0e6","Type":"ContainerStarted","Data":"dd3c45a5901b7b385fd8e15eb539436854fdef3c6ba5b64046e2817b6fec62ef"} Jul 22 00:31:18.949738 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:18.949706 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" event={"ID":"e0a1eb7a-b0dc-4804-975f-c601526d229e","Type":"ContainerStarted","Data":"48daa8950170c5d6a2e4aebf80e5bdab6b907cf3ddcf8fdb334afdac1e7c9eff"} Jul 22 00:31:18.950172 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:18.949752 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" event={"ID":"e0a1eb7a-b0dc-4804-975f-c601526d229e","Type":"ContainerStarted","Data":"efb218e33ab871bcfb093d92184eb583b77e08be40f9108acace78225ad31f49"} Jul 22 00:31:18.950172 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:18.949829 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:31:18.969642 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:18.969549 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" podStartSLOduration=286.969534419 podStartE2EDuration="4m46.969534419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:18.968224252 +0000 UTC m=+287.294296455" watchObservedRunningTime="2026-07-22 00:31:18.969534419 +0000 UTC m=+287.295606620" Jul 22 00:31:19.417207 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:19.417170 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-dqz2j"] Jul 22 00:31:19.420408 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:19.420387 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-dqz2j" Jul 22 00:31:19.423124 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:19.423099 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 00:31:19.423257 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:19.423182 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lttng\"" Jul 22 00:31:19.423319 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:19.423257 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 00:31:19.430847 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:19.430826 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-dqz2j"] Jul 22 00:31:19.508721 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:19.508681 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:19.553714 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:19.553685 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55bc5\" (UniqueName: \"kubernetes.io/projected/ae9fa646-a31d-473d-ab86-a537257cfe17-kube-api-access-55bc5\") pod \"downloads-5bc67dcd46-dqz2j\" (UID: \"ae9fa646-a31d-473d-ab86-a537257cfe17\") " pod="openshift-console/downloads-5bc67dcd46-dqz2j" Jul 22 00:31:19.613634 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:19.613582 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:19.654821 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:19.654788 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55bc5\" (UniqueName: \"kubernetes.io/projected/ae9fa646-a31d-473d-ab86-a537257cfe17-kube-api-access-55bc5\") pod \"downloads-5bc67dcd46-dqz2j\" (UID: \"ae9fa646-a31d-473d-ab86-a537257cfe17\") " pod="openshift-console/downloads-5bc67dcd46-dqz2j" Jul 22 00:31:19.664442 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:19.664419 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55bc5\" (UniqueName: \"kubernetes.io/projected/ae9fa646-a31d-473d-ab86-a537257cfe17-kube-api-access-55bc5\") pod \"downloads-5bc67dcd46-dqz2j\" (UID: \"ae9fa646-a31d-473d-ab86-a537257cfe17\") " pod="openshift-console/downloads-5bc67dcd46-dqz2j" Jul 22 00:31:19.732229 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:19.732160 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-dqz2j" Jul 22 00:31:19.955158 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:19.955049 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" event={"ID":"e97ba97d-b165-4bd4-9a46-d8ac502f5d9d","Type":"ContainerStarted","Data":"efe6f8a6c9aa6b8a3e612f07f722235a59c8cfa75df0484273002c500716e278"} Jul 22 00:31:19.972931 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:19.972879 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-zmtpr" podStartSLOduration=273.730903801 podStartE2EDuration="4m34.972859668s" totalImagesPullingTime="1.241955867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:19.97163226 +0000 UTC m=+288.297704462" watchObservedRunningTime="2026-07-22 00:31:19.972859668 +0000 UTC m=+288.298931872" Jul 22 00:31:19.985938 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:19.985914 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-dqz2j"] Jul 22 00:31:19.989560 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:31:19.989531 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9fa646_a31d_473d_ab86_a537257cfe17.slice/crio-0c6a86d91467756262c6bcfab281912f3c1c9c77a8ce61eb5120225d53063e2c WatchSource:0}: Error finding container 0c6a86d91467756262c6bcfab281912f3c1c9c77a8ce61eb5120225d53063e2c: Status 404 returned error can't find the container with id 0c6a86d91467756262c6bcfab281912f3c1c9c77a8ce61eb5120225d53063e2c Jul 22 00:31:20.072704 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:20.072671 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:20.173534 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:20.173505 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:20.765049 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:20.765010 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:31:20.765049 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:20.765052 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:31:20.765233 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:20.765105 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-tbc4j\" (UID: \"a7accd6c-8f60-4679-9ae1-331f59788c1b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:31:20.765813 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:20.765765 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-service-ca-bundle\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:31:20.767502 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:20.767474 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09532874-5a40-44cd-b95b-1b3e9ad8ecfd-metrics-certs\") pod \"router-default-7584584bbd-c77b7\" (UID: \"09532874-5a40-44cd-b95b-1b3e9ad8ecfd\") " pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:31:20.767587 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:20.767555 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7accd6c-8f60-4679-9ae1-331f59788c1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-tbc4j\" (UID: \"a7accd6c-8f60-4679-9ae1-331f59788c1b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:31:20.951128 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:20.951100 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" Jul 22 00:31:20.954959 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:20.954927 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:31:20.960543 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:20.960369 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cdzlx" event={"ID":"0f69f381-426d-422c-a9df-a007c2e4d0e6","Type":"ContainerStarted","Data":"fafa0a6c04745033423d7f1fb870d6fd4590ca64f21c7dd3c861876b03da7de8"} Jul 22 00:31:20.960543 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:20.960429 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cdzlx" event={"ID":"0f69f381-426d-422c-a9df-a007c2e4d0e6","Type":"ContainerStarted","Data":"e08f9c35a706ce4441cd44dffc99f741ea99a70d6e28644da348c2243d692e04"} Jul 22 00:31:20.960543 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:20.960460 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cdzlx" Jul 22 00:31:20.961857 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:20.961808 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-dqz2j" event={"ID":"ae9fa646-a31d-473d-ab86-a537257cfe17","Type":"ContainerStarted","Data":"0c6a86d91467756262c6bcfab281912f3c1c9c77a8ce61eb5120225d53063e2c"} Jul 22 00:31:20.977383 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:20.977340 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cdzlx" podStartSLOduration=252.042000985 podStartE2EDuration="4m13.97732552s" totalImagesPullingTime="1.935324535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:20.975188862 +0000 UTC m=+289.301261068" watchObservedRunningTime="2026-07-22 00:31:20.97732552 +0000 UTC m=+289.303397721" Jul 22 00:31:21.080547 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:21.080518 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j"] Jul 22 00:31:21.083520 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:31:21.083489 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7accd6c_8f60_4679_9ae1_331f59788c1b.slice/crio-60e8cf2b54d7d3ec9777db79ec44ff0ecf0aa17b42f2c70f9fa79493daa77385 WatchSource:0}: Error finding container 60e8cf2b54d7d3ec9777db79ec44ff0ecf0aa17b42f2c70f9fa79493daa77385: Status 404 returned error can't find the container with id 60e8cf2b54d7d3ec9777db79ec44ff0ecf0aa17b42f2c70f9fa79493daa77385 Jul 22 00:31:21.102514 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:21.102481 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7584584bbd-c77b7"] Jul 22 00:31:21.106017 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:31:21.105993 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09532874_5a40_44cd_b95b_1b3e9ad8ecfd.slice/crio-338802872abb78be9b62008d26f0f18b0f521a566c4ed65782ae67148e93abf8 WatchSource:0}: Error finding container 338802872abb78be9b62008d26f0f18b0f521a566c4ed65782ae67148e93abf8: Status 404 returned error can't find the container with id 338802872abb78be9b62008d26f0f18b0f521a566c4ed65782ae67148e93abf8 Jul 22 00:31:21.750524 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:21.750476 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:21.799403 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:21.799369 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:21.968307 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:21.968274 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7584584bbd-c77b7" event={"ID":"09532874-5a40-44cd-b95b-1b3e9ad8ecfd","Type":"ContainerStarted","Data":"b62aeba578017fa9c023c3488bef9c60ebd96fa681c540f3ed34703898b349b6"} Jul 22 00:31:21.968763 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:21.968366 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7584584bbd-c77b7" event={"ID":"09532874-5a40-44cd-b95b-1b3e9ad8ecfd","Type":"ContainerStarted","Data":"338802872abb78be9b62008d26f0f18b0f521a566c4ed65782ae67148e93abf8"} Jul 22 00:31:21.969556 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:21.969524 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" event={"ID":"a7accd6c-8f60-4679-9ae1-331f59788c1b","Type":"ContainerStarted","Data":"60e8cf2b54d7d3ec9777db79ec44ff0ecf0aa17b42f2c70f9fa79493daa77385"} Jul 22 00:31:21.986877 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:21.986819 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7584584bbd-c77b7" podStartSLOduration=33.986800843 podStartE2EDuration="33.986800843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:21.986140791 +0000 UTC m=+290.312213019" watchObservedRunningTime="2026-07-22 00:31:21.986800843 +0000 UTC m=+290.312873045" Jul 22 00:31:22.955545 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:22.955511 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:31:22.958390 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:22.958356 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:31:22.974258 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:22.974234 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:31:22.975397 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:22.975377 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7584584bbd-c77b7" Jul 22 00:31:23.339406 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:23.339368 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:23.436903 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:23.436847 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:23.508860 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:23.508820 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:23.652595 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:23.652518 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:23.979398 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:23.979317 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" event={"ID":"a7accd6c-8f60-4679-9ae1-331f59788c1b","Type":"ContainerStarted","Data":"8914cdc3502e325292a9bcfddf7c4d983951be9fa8c00e60bbfa78409ec45a9a"} Jul 22 00:31:23.995745 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:23.995698 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-tbc4j" podStartSLOduration=34.091985879 podStartE2EDuration="35.995681971s" totalImagesPullingTime="1.903696092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:23.994937857 +0000 UTC m=+292.321010060" watchObservedRunningTime="2026-07-22 00:31:23.995681971 +0000 UTC m=+292.321754174" Jul 22 00:31:25.806107 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:25.806075 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:31:25.808219 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:25.808193 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a087a509-8906-4b67-a39d-bf24cd20c7da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-96fng\" (UID: \"a087a509-8906-4b67-a39d-bf24cd20c7da\") " pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:31:26.097022 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.096954 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-96fng" Jul 22 00:31:26.232302 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.232269 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-96fng"] Jul 22 00:31:26.237141 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:31:26.236963 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda087a509_8906_4b67_a39d_bf24cd20c7da.slice/crio-40ec5727369b3509dd10a38c66adaeda624c1694b037f46dadd2801c4433bca6 WatchSource:0}: Error finding container 40ec5727369b3509dd10a38c66adaeda624c1694b037f46dadd2801c4433bca6: Status 404 returned error can't find the container with id 40ec5727369b3509dd10a38c66adaeda624c1694b037f46dadd2801c4433bca6 Jul 22 00:31:26.387716 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.387625 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:26.393653 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.393623 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bc4c9b4c8-7tqc5"] Jul 22 00:31:26.397982 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.397959 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.400527 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.400508 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 00:31:26.400650 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.400580 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 00:31:26.400814 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.400797 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 00:31:26.400898 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.400844 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t9krq\"" Jul 22 00:31:26.401657 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.401623 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 00:31:26.401657 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.401640 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 00:31:26.407961 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.407940 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bc4c9b4c8-7tqc5"] Jul 22 00:31:26.457228 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.457200 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:26.511069 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.511044 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-oauth-config\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.511193 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.511075 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-config\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.511193 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.511176 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-oauth-serving-cert\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.511292 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.511202 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-serving-cert\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.511292 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.511224 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-service-ca\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.511292 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.511238 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svlk5\" (UniqueName: \"kubernetes.io/projected/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-kube-api-access-svlk5\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.518083 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.518055 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:26.575428 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.575396 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:26.611920 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.611893 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-oauth-config\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.612030 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.611932 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-config\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.612030 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.612010 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-oauth-serving-cert\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.612130 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.612039 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-serving-cert\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.612130 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.612072 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-service-ca\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.612130 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.612098 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svlk5\" (UniqueName: \"kubernetes.io/projected/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-kube-api-access-svlk5\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.612768 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.612745 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-config\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.613057 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.613033 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-service-ca\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.613133 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.613041 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-oauth-serving-cert\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.614647 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.614621 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-oauth-config\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.615144 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.615123 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-serving-cert\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.619636 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.619616 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svlk5\" (UniqueName: \"kubernetes.io/projected/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-kube-api-access-svlk5\") pod \"console-bc4c9b4c8-7tqc5\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.681106 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.681041 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:26.709575 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.709553 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:26.886216 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.886188 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bc4c9b4c8-7tqc5"] Jul 22 00:31:26.889497 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:31:26.889466 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2409b4_9efc_4d67_af2c_6a67bf3b60c8.slice/crio-d6ab2a330a34cd6b7210d92869d9403f0d1e4c69ace407fd9a177244edbc203a WatchSource:0}: Error finding container d6ab2a330a34cd6b7210d92869d9403f0d1e4c69ace407fd9a177244edbc203a: Status 404 returned error can't find the container with id d6ab2a330a34cd6b7210d92869d9403f0d1e4c69ace407fd9a177244edbc203a Jul 22 00:31:26.997543 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.997419 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc4c9b4c8-7tqc5" event={"ID":"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8","Type":"ContainerStarted","Data":"d6ab2a330a34cd6b7210d92869d9403f0d1e4c69ace407fd9a177244edbc203a"} Jul 22 00:31:26.999533 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:26.999501 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-96fng" event={"ID":"a087a509-8906-4b67-a39d-bf24cd20c7da","Type":"ContainerStarted","Data":"40ec5727369b3509dd10a38c66adaeda624c1694b037f46dadd2801c4433bca6"} Jul 22 00:31:28.112916 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:28.112868 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6q2tc" podUID="d55898f6-dc0e-408a-99cc-9883ae591ac5" Jul 22 00:31:28.345876 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:28.345832 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:28.924555 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:28.924458 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:29.796822 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:29.796765 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:29.846653 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:29.846617 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:30.972115 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:30.972066 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cdzlx" Jul 22 00:31:32.181159 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:32.180959 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:32.234448 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:32.234413 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:33.185201 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.185161 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:33.635136 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.635101 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:33.847753 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.847717 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:33.871752 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.871714 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz"] Jul 22 00:31:33.875308 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.875290 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" Jul 22 00:31:33.877893 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.877871 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 22 00:31:33.877893 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.877887 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bsbk4\"" Jul 22 00:31:33.878943 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.878880 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 22 00:31:33.878943 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.878928 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 00:31:33.884513 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.884492 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz"] Jul 22 00:31:33.899330 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.899277 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v7qfb"] Jul 22 00:31:33.902507 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.902488 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:33.904912 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.904886 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 00:31:33.905024 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.904893 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-49khf\"" Jul 22 00:31:33.905236 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.905204 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 00:31:33.905331 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.905259 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 00:31:33.935564 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.935498 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:33.976882 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.976861 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f43e233f-b908-4f2c-aeaa-d9be509395b4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-f4pmz\" (UID: \"f43e233f-b908-4f2c-aeaa-d9be509395b4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" Jul 22 00:31:33.976998 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.976896 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f43e233f-b908-4f2c-aeaa-d9be509395b4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-f4pmz\" (UID: \"f43e233f-b908-4f2c-aeaa-d9be509395b4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" Jul 22 00:31:33.976998 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.976935 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f43e233f-b908-4f2c-aeaa-d9be509395b4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-f4pmz\" (UID: \"f43e233f-b908-4f2c-aeaa-d9be509395b4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" Jul 22 00:31:33.976998 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:33.976960 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h22k\" (UniqueName: \"kubernetes.io/projected/f43e233f-b908-4f2c-aeaa-d9be509395b4-kube-api-access-6h22k\") pod \"openshift-state-metrics-79f6958cdf-f4pmz\" (UID: \"f43e233f-b908-4f2c-aeaa-d9be509395b4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" Jul 22 00:31:34.077788 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.077739 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0609636-2778-4018-aed2-68861437faa2-sys\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.077935 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.077805 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c0609636-2778-4018-aed2-68861437faa2-root\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.077935 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.077839 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c0609636-2778-4018-aed2-68861437faa2-node-exporter-tls\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.077935 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.077874 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c0609636-2778-4018-aed2-68861437faa2-node-exporter-textfile\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.078094 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.077931 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c0609636-2778-4018-aed2-68861437faa2-node-exporter-wtmp\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.078094 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.078042 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f43e233f-b908-4f2c-aeaa-d9be509395b4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-f4pmz\" (UID: \"f43e233f-b908-4f2c-aeaa-d9be509395b4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" Jul 22 00:31:34.078201 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.078090 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0609636-2778-4018-aed2-68861437faa2-metrics-client-ca\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.078201 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.078115 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrzh2\" (UniqueName: \"kubernetes.io/projected/c0609636-2778-4018-aed2-68861437faa2-kube-api-access-lrzh2\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.078201 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.078145 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f43e233f-b908-4f2c-aeaa-d9be509395b4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-f4pmz\" (UID: \"f43e233f-b908-4f2c-aeaa-d9be509395b4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" Jul 22 00:31:34.078201 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.078179 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0609636-2778-4018-aed2-68861437faa2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.078356 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.078211 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f43e233f-b908-4f2c-aeaa-d9be509395b4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-f4pmz\" (UID: \"f43e233f-b908-4f2c-aeaa-d9be509395b4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" Jul 22 00:31:34.078356 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.078232 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h22k\" (UniqueName: \"kubernetes.io/projected/f43e233f-b908-4f2c-aeaa-d9be509395b4-kube-api-access-6h22k\") pod \"openshift-state-metrics-79f6958cdf-f4pmz\" (UID: \"f43e233f-b908-4f2c-aeaa-d9be509395b4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" Jul 22 00:31:34.078356 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.078275 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c0609636-2778-4018-aed2-68861437faa2-node-exporter-accelerators-collector-config\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.078356 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:34.078292 2567 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 22 00:31:34.078356 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:34.078355 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f43e233f-b908-4f2c-aeaa-d9be509395b4-openshift-state-metrics-tls podName:f43e233f-b908-4f2c-aeaa-d9be509395b4 nodeName:}" failed. No retries permitted until 2026-07-22 00:31:34.578336348 +0000 UTC m=+302.904408534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f43e233f-b908-4f2c-aeaa-d9be509395b4-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-f4pmz" (UID: "f43e233f-b908-4f2c-aeaa-d9be509395b4") : secret "openshift-state-metrics-tls" not found Jul 22 00:31:34.079100 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.079073 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f43e233f-b908-4f2c-aeaa-d9be509395b4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-f4pmz\" (UID: \"f43e233f-b908-4f2c-aeaa-d9be509395b4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" Jul 22 00:31:34.080654 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.080630 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f43e233f-b908-4f2c-aeaa-d9be509395b4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-f4pmz\" (UID: \"f43e233f-b908-4f2c-aeaa-d9be509395b4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" Jul 22 00:31:34.089089 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.089070 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h22k\" (UniqueName: \"kubernetes.io/projected/f43e233f-b908-4f2c-aeaa-d9be509395b4-kube-api-access-6h22k\") pod \"openshift-state-metrics-79f6958cdf-f4pmz\" (UID: \"f43e233f-b908-4f2c-aeaa-d9be509395b4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" Jul 22 00:31:34.133740 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.133709 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:34.178735 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.178676 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0609636-2778-4018-aed2-68861437faa2-sys\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.178735 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.178722 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c0609636-2778-4018-aed2-68861437faa2-root\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.178922 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.178752 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c0609636-2778-4018-aed2-68861437faa2-node-exporter-tls\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.178922 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.178853 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c0609636-2778-4018-aed2-68861437faa2-root\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.178922 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:34.178890 2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 00:31:34.178922 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.178905 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0609636-2778-4018-aed2-68861437faa2-sys\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.179112 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.178932 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c0609636-2778-4018-aed2-68861437faa2-node-exporter-textfile\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.179112 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:34.178947 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0609636-2778-4018-aed2-68861437faa2-node-exporter-tls podName:c0609636-2778-4018-aed2-68861437faa2 nodeName:}" failed. No retries permitted until 2026-07-22 00:31:34.678928417 +0000 UTC m=+303.005000598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c0609636-2778-4018-aed2-68861437faa2-node-exporter-tls") pod "node-exporter-v7qfb" (UID: "c0609636-2778-4018-aed2-68861437faa2") : secret "node-exporter-tls" not found Jul 22 00:31:34.179112 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.178976 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c0609636-2778-4018-aed2-68861437faa2-node-exporter-wtmp\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.179112 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.179040 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0609636-2778-4018-aed2-68861437faa2-metrics-client-ca\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.179112 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.179064 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrzh2\" (UniqueName: \"kubernetes.io/projected/c0609636-2778-4018-aed2-68861437faa2-kube-api-access-lrzh2\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.179317 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.179127 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0609636-2778-4018-aed2-68861437faa2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.179317 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.179188 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c0609636-2778-4018-aed2-68861437faa2-node-exporter-wtmp\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.179317 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.179196 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c0609636-2778-4018-aed2-68861437faa2-node-exporter-accelerators-collector-config\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.179317 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.179261 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c0609636-2778-4018-aed2-68861437faa2-node-exporter-textfile\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.179755 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.179733 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0609636-2778-4018-aed2-68861437faa2-metrics-client-ca\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.179892 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.179832 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c0609636-2778-4018-aed2-68861437faa2-node-exporter-accelerators-collector-config\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.181629 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.181607 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0609636-2778-4018-aed2-68861437faa2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.187047 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.187025 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrzh2\" (UniqueName: \"kubernetes.io/projected/c0609636-2778-4018-aed2-68861437faa2-kube-api-access-lrzh2\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.241093 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.241061 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:34.383826 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.383764 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:34.582405 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.582374 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f43e233f-b908-4f2c-aeaa-d9be509395b4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-f4pmz\" (UID: \"f43e233f-b908-4f2c-aeaa-d9be509395b4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" Jul 22 00:31:34.585071 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.585044 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f43e233f-b908-4f2c-aeaa-d9be509395b4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-f4pmz\" (UID: \"f43e233f-b908-4f2c-aeaa-d9be509395b4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" Jul 22 00:31:34.683299 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.683265 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c0609636-2778-4018-aed2-68861437faa2-node-exporter-tls\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:34.683468 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:34.683446 2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 00:31:34.683542 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:31:34.683530 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0609636-2778-4018-aed2-68861437faa2-node-exporter-tls podName:c0609636-2778-4018-aed2-68861437faa2 nodeName:}" failed. No retries permitted until 2026-07-22 00:31:35.683506903 +0000 UTC m=+304.009579088 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c0609636-2778-4018-aed2-68861437faa2-node-exporter-tls") pod "node-exporter-v7qfb" (UID: "c0609636-2778-4018-aed2-68861437faa2") : secret "node-exporter-tls" not found Jul 22 00:31:34.785907 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.785880 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" Jul 22 00:31:34.952389 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.952305 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 00:31:34.956233 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.956205 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:34.958743 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.958721 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 00:31:34.958904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.958859 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 00:31:34.958994 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.958941 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 00:31:34.958994 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.958969 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 00:31:34.959169 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.959141 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 00:31:34.959245 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.959236 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 00:31:34.959310 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.959236 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 00:31:34.959386 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.959337 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 00:31:34.959521 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.959500 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zsft8\"" Jul 22 00:31:34.964286 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.964266 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 00:31:34.971571 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:34.971551 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 00:31:35.086730 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.086694 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cef264aa-037a-482c-ba8e-dbca38f79a27-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.086906 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.086767 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cef264aa-037a-482c-ba8e-dbca38f79a27-config-out\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.086906 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.086847 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.086906 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.086876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.086906 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.086900 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cef264aa-037a-482c-ba8e-dbca38f79a27-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.087094 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.086995 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cef264aa-037a-482c-ba8e-dbca38f79a27-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.087094 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.087030 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-web-config\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.087215 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.087110 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.087215 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.087160 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.087215 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.087186 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jphxm\" (UniqueName: \"kubernetes.io/projected/cef264aa-037a-482c-ba8e-dbca38f79a27-kube-api-access-jphxm\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.087347 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.087224 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cef264aa-037a-482c-ba8e-dbca38f79a27-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.087347 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.087274 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.087347 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.087297 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-config-volume\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.188340 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.188314 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cef264aa-037a-482c-ba8e-dbca38f79a27-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.188771 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.188368 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cef264aa-037a-482c-ba8e-dbca38f79a27-config-out\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.188771 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.188416 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.188771 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.188443 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.188771 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.188470 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cef264aa-037a-482c-ba8e-dbca38f79a27-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.188771 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.188511 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cef264aa-037a-482c-ba8e-dbca38f79a27-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.188771 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.188532 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-web-config\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.188771 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.188557 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.188771 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.188584 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.188771 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.188609 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jphxm\" (UniqueName: \"kubernetes.io/projected/cef264aa-037a-482c-ba8e-dbca38f79a27-kube-api-access-jphxm\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.188771 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.188635 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cef264aa-037a-482c-ba8e-dbca38f79a27-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.188771 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.188671 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.188771 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.188701 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-config-volume\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.189397 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.189155 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cef264aa-037a-482c-ba8e-dbca38f79a27-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.189397 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.189279 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cef264aa-037a-482c-ba8e-dbca38f79a27-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.190610 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.190579 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cef264aa-037a-482c-ba8e-dbca38f79a27-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.191936 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.191908 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cef264aa-037a-482c-ba8e-dbca38f79a27-config-out\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.192043 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.191998 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.192247 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.192225 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cef264aa-037a-482c-ba8e-dbca38f79a27-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.193463 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.193437 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.193575 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.193488 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-web-config\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.193880 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.193854 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.194190 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.194163 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.194283 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.194171 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.194465 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.194445 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-config-volume\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.197106 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.197083 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jphxm\" (UniqueName: \"kubernetes.io/projected/cef264aa-037a-482c-ba8e-dbca38f79a27-kube-api-access-jphxm\") pod \"alertmanager-main-0\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.267722 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.267695 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:31:35.693110 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.693024 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c0609636-2778-4018-aed2-68861437faa2-node-exporter-tls\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:35.695410 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.695377 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c0609636-2778-4018-aed2-68861437faa2-node-exporter-tls\") pod \"node-exporter-v7qfb\" (UID: \"c0609636-2778-4018-aed2-68861437faa2\") " pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:35.713442 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:35.713417 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v7qfb" Jul 22 00:31:37.186762 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:37.186724 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:37.600144 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:37.600111 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:37.837840 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:37.837801 2567 patch_prober.go:28] interesting pod/image-registry-58fcb4cbcd-j5fmf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 22 00:31:37.837997 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:37.837859 2567 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" podUID="e0a1eb7a-b0dc-4804-975f-c601526d229e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 00:31:38.663668 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:38.663625 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:38.722033 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:38.721922 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:39.052761 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:39.052729 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bc4c9b4c8-7tqc5"] Jul 22 00:31:39.095189 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:39.095146 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:39.455301 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:39.455172 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:39.512305 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:39.512254 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:39.960268 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:39.960239 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:31:40.023504 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:40.023466 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:31:40.023708 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:40.023689 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:31:40.034853 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:40.034829 2567 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 00:31:40.040866 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:31:40.040747 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0609636_2778_4018_aed2_68861437faa2.slice/crio-ed72c704058c96845d3dbacecb59aabf2a823973d2e15aca75e7e3be9037e41e WatchSource:0}: Error finding container ed72c704058c96845d3dbacecb59aabf2a823973d2e15aca75e7e3be9037e41e: Status 404 returned error can't find the container with id ed72c704058c96845d3dbacecb59aabf2a823973d2e15aca75e7e3be9037e41e Jul 22 00:31:40.047903 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:40.046297 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 00:31:40.062842 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:40.062773 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v7qfb" event={"ID":"c0609636-2778-4018-aed2-68861437faa2","Type":"ContainerStarted","Data":"ed72c704058c96845d3dbacecb59aabf2a823973d2e15aca75e7e3be9037e41e"} Jul 22 00:31:40.167896 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:40.167855 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 00:31:40.171495 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:31:40.171453 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef264aa_037a_482c_ba8e_dbca38f79a27.slice/crio-b03e265da2b977b8a88134a85b4fe2e2131ff224b9de73765db733049e432adc WatchSource:0}: Error finding container b03e265da2b977b8a88134a85b4fe2e2131ff224b9de73765db733049e432adc: Status 404 returned error can't find the container with id b03e265da2b977b8a88134a85b4fe2e2131ff224b9de73765db733049e432adc Jul 22 00:31:40.185499 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:40.185424 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz"] Jul 22 00:31:40.276597 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:40.276494 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:40.376414 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:40.376352 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:41.069484 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:41.069444 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc4c9b4c8-7tqc5" event={"ID":"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8","Type":"ContainerStarted","Data":"4a55fbb2ea97fef10271d60e5dc2e85a6e29508cf1f917ec2ca3abaa8616b891"} Jul 22 00:31:41.071698 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:41.071664 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" event={"ID":"f43e233f-b908-4f2c-aeaa-d9be509395b4","Type":"ContainerStarted","Data":"4cd7774e0d8558cf22032e4914a8a8308cbd801601a5f9e5c4b144e1b805d96f"} Jul 22 00:31:41.071839 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:41.071730 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" event={"ID":"f43e233f-b908-4f2c-aeaa-d9be509395b4","Type":"ContainerStarted","Data":"1093053d189617780d3913723f6fb2a82cd9303333cd767835a4b2abc40b5148"} Jul 22 00:31:41.071839 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:41.071762 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" event={"ID":"f43e233f-b908-4f2c-aeaa-d9be509395b4","Type":"ContainerStarted","Data":"0c94a89a23cc3aaefd43f03a0805fc5dc0c488b5b7f91c3a1b709ee218a1b87d"} Jul 22 00:31:41.073448 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:41.073418 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-96fng" event={"ID":"a087a509-8906-4b67-a39d-bf24cd20c7da","Type":"ContainerStarted","Data":"1d7287a8abb15a894b96deb74191936a695f08d02b53dc0e8cdc8392044d1c55"} Jul 22 00:31:41.074829 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:41.074763 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cef264aa-037a-482c-ba8e-dbca38f79a27","Type":"ContainerStarted","Data":"b03e265da2b977b8a88134a85b4fe2e2131ff224b9de73765db733049e432adc"} Jul 22 00:31:41.076717 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:41.076663 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-dqz2j" event={"ID":"ae9fa646-a31d-473d-ab86-a537257cfe17","Type":"ContainerStarted","Data":"ac503d6cd982898c685a0e1f2d6b1d93ea953896f5459d7f2133cf28cacb6e4d"} Jul 22 00:31:41.076857 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:41.076825 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-dqz2j" Jul 22 00:31:41.084576 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:41.084555 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-dqz2j" Jul 22 00:31:41.087070 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:41.087029 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bc4c9b4c8-7tqc5" podStartSLOduration=1.993289386 podStartE2EDuration="15.087013051s" totalImagesPullingTime="13.093723665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:41.086412685 +0000 UTC m=+309.412484889" watchObservedRunningTime="2026-07-22 00:31:41.087013051 +0000 UTC m=+309.413085253" Jul 22 00:31:41.102719 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:41.102602 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-dqz2j" podStartSLOduration=2.032520588 podStartE2EDuration="22.102584326s" totalImagesPullingTime="20.070063738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:41.100664389 +0000 UTC m=+309.426736592" watchObservedRunningTime="2026-07-22 00:31:41.102584326 +0000 UTC m=+309.428656529" Jul 22 00:31:41.958068 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:41.957369 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58fcb4cbcd-j5fmf"] Jul 22 00:31:42.046342 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:42.046305 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:42.086599 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:42.086560 2567 generic.go:356] "Generic (PLEG): container finished" podID="c0609636-2778-4018-aed2-68861437faa2" containerID="e89b5b331bf8824f820391fdf1267e9aeb9ee39898dcff8ecb8f8cb11cdb7c6b" exitCode=0 Jul 22 00:31:42.088031 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:42.087694 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v7qfb" event={"ID":"c0609636-2778-4018-aed2-68861437faa2","Type":"ContainerDied","Data":"e89b5b331bf8824f820391fdf1267e9aeb9ee39898dcff8ecb8f8cb11cdb7c6b"} Jul 22 00:31:42.109936 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:42.109894 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:42.114876 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:42.114853 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:31:42.217629 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:42.217496 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:42.319506 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:42.319414 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:43.096660 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:43.096513 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v7qfb" event={"ID":"c0609636-2778-4018-aed2-68861437faa2","Type":"ContainerStarted","Data":"d670b840e6060ec6a4f22b440b70def151572f25a74870a2d105dfb3143f6b8d"} Jul 22 00:31:43.096660 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:43.096612 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v7qfb" event={"ID":"c0609636-2778-4018-aed2-68861437faa2","Type":"ContainerStarted","Data":"f52e21e21e1921e729988ce198b234810a2030552b6c214a810d9bbe09c5329c"} Jul 22 00:31:43.117804 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:43.116666 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v7qfb" podStartSLOduration=9.062974927 podStartE2EDuration="10.116645446s" totalImagesPullingTime="971.712722ms" totalInitContainerRuntime="81.957797ms" isStatefulPod=true podCreationTimestamp="2026-07-22 00:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:43.115649352 +0000 UTC m=+311.441721570" watchObservedRunningTime="2026-07-22 00:31:43.116645446 +0000 UTC m=+311.442717649" Jul 22 00:31:44.138523 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:44.138488 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:44.190252 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:44.190214 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:46.710870 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:46.710845 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:31:47.116735 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:47.116709 2567 generic.go:356] "Generic (PLEG): container finished" podID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerID="42c35dc186e02e76fccaa16d01116fb477681cffb70c81be5aeba5ccf6f92783" exitCode=0 Jul 22 00:31:47.116867 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:47.116809 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cef264aa-037a-482c-ba8e-dbca38f79a27","Type":"ContainerDied","Data":"42c35dc186e02e76fccaa16d01116fb477681cffb70c81be5aeba5ccf6f92783"} Jul 22 00:31:47.119538 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:47.119513 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" event={"ID":"f43e233f-b908-4f2c-aeaa-d9be509395b4","Type":"ContainerStarted","Data":"242e748f2f1d73427084b1b80900f6722acac3271ec44f39ede10139f0661715"} Jul 22 00:31:47.157679 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:47.157634 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-f4pmz" podStartSLOduration=7.84741589 podStartE2EDuration="14.157614037s" totalImagesPullingTime="6.310198147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:47.156981124 +0000 UTC m=+315.483053347" watchObservedRunningTime="2026-07-22 00:31:47.157614037 +0000 UTC m=+315.483686240" Jul 22 00:31:48.125921 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:48.125881 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-96fng" event={"ID":"a087a509-8906-4b67-a39d-bf24cd20c7da","Type":"ContainerStarted","Data":"04a3befa1f7c7106810641c97edfcdf33686e1204b24a018895c0f9c99c81f69"} Jul 22 00:31:50.037234 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:50.037193 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:50.088304 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:50.088260 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:50.234019 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:50.233983 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:51.788414 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:51.788380 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:51.848332 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:51.848294 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:52.125244 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:52.125168 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:52.177652 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:52.177622 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:53.151384 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:53.151355 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-96fng" event={"ID":"a087a509-8906-4b67-a39d-bf24cd20c7da","Type":"ContainerStarted","Data":"00c3610aa3804170b1f197fd9268a6a17cddd2274a6033ea9b9b9dcf52a96a80"} Jul 22 00:31:53.154281 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:53.154256 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cef264aa-037a-482c-ba8e-dbca38f79a27","Type":"ContainerStarted","Data":"c98571b4834b3152d4991c4bd843c576703908eb711cb01758bb18cd74fdeebb"} Jul 22 00:31:53.154419 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:53.154320 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cef264aa-037a-482c-ba8e-dbca38f79a27","Type":"ContainerStarted","Data":"aa6d7e10a676e4ff68d39d4171e2f65a5b428a20479c6512574975cb38ab1302"} Jul 22 00:31:53.154419 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:53.154357 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cef264aa-037a-482c-ba8e-dbca38f79a27","Type":"ContainerStarted","Data":"8735173f284503ea25529ae96d2257f8cb3ca625fba9afa4eb6a69ee8ce94b03"} Jul 22 00:31:53.154419 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:53.154396 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cef264aa-037a-482c-ba8e-dbca38f79a27","Type":"ContainerStarted","Data":"7513a18cd70599e687216678e3569b5313b648ad69020b1ea582cd753fbf0693"} Jul 22 00:31:53.154551 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:53.154430 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cef264aa-037a-482c-ba8e-dbca38f79a27","Type":"ContainerStarted","Data":"b8fadde9c9aa8a92a1949afe1522c1c364e66c9e17d84f9c18ded3e8cdb3777b"} Jul 22 00:31:53.170180 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:53.170130 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-96fng" podStartSLOduration=34.350934252 podStartE2EDuration="1m0.170115677s" totalImagesPullingTime="25.819181425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:30:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:53.168724713 +0000 UTC m=+321.494796933" watchObservedRunningTime="2026-07-22 00:31:53.170115677 +0000 UTC m=+321.496187879" Jul 22 00:31:55.170963 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:55.170929 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cef264aa-037a-482c-ba8e-dbca38f79a27","Type":"ContainerStarted","Data":"b0d8c1b3e651385ae6a2aede999e79c839aef4514be75bd7d73f27694f0b8eeb"} Jul 22 00:31:55.201799 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:55.201735 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=7.80363001 podStartE2EDuration="21.201719094s" totalImagesPullingTime="13.370607779s" totalInitContainerRuntime="27.481305ms" isStatefulPod=false podCreationTimestamp="2026-07-22 00:31:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:55.199879129 +0000 UTC m=+323.525951331" watchObservedRunningTime="2026-07-22 00:31:55.201719094 +0000 UTC m=+323.527791299" Jul 22 00:31:58.436196 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:58.436157 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:58.534784 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:58.534755 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:31:59.187681 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:59.187654 2567 generic.go:356] "Generic (PLEG): container finished" podID="eb290190-a2f9-4fd6-8267-b395887f767b" containerID="6a54d18f04fe291c65552c4dec4514f6f5822176c216534a663e2d7b9c8d4d41" exitCode=0 Jul 22 00:31:59.187911 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:59.187730 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-sdhl6" event={"ID":"eb290190-a2f9-4fd6-8267-b395887f767b","Type":"ContainerDied","Data":"6a54d18f04fe291c65552c4dec4514f6f5822176c216534a663e2d7b9c8d4d41"} Jul 22 00:31:59.188070 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:59.188056 2567 scope.go:117] "RemoveContainer" containerID="6a54d18f04fe291c65552c4dec4514f6f5822176c216534a663e2d7b9c8d4d41" Jul 22 00:31:59.189181 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:59.189161 2567 generic.go:356] "Generic (PLEG): container finished" podID="f0b40554-c78f-463e-bce6-b4ac0b2b1c51" containerID="fdb15bcd48ccdd6fb1160819072ff5c7772828c75a733d38efb67bb867c18e01" exitCode=0 Jul 22 00:31:59.189262 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:59.189201 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" event={"ID":"f0b40554-c78f-463e-bce6-b4ac0b2b1c51","Type":"ContainerDied","Data":"fdb15bcd48ccdd6fb1160819072ff5c7772828c75a733d38efb67bb867c18e01"} Jul 22 00:31:59.189421 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:31:59.189410 2567 scope.go:117] "RemoveContainer" containerID="fdb15bcd48ccdd6fb1160819072ff5c7772828c75a733d38efb67bb867c18e01" Jul 22 00:32:00.119805 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:00.119716 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cdzlx_0f69f381-426d-422c-a9df-a007c2e4d0e6/dns/0.log" Jul 22 00:32:00.194586 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:00.194556 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kfnl7" event={"ID":"f0b40554-c78f-463e-bce6-b4ac0b2b1c51","Type":"ContainerStarted","Data":"1676ea642a623af9364fad25f44395829a5188327e2379fe3dbc4c2e8fb43131"} Jul 22 00:32:00.196184 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:00.196161 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-sdhl6" event={"ID":"eb290190-a2f9-4fd6-8267-b395887f767b","Type":"ContainerStarted","Data":"ae8862248787fa9be5fb255384c0ce60598342b4fbd89b18e789191eb7179c74"} Jul 22 00:32:00.315353 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:00.315328 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cdzlx_0f69f381-426d-422c-a9df-a007c2e4d0e6/kube-rbac-proxy/0.log" Jul 22 00:32:00.715078 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:00.715052 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kxz6k_c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5/dns-node-resolver/0.log" Jul 22 00:32:00.852675 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:00.852636 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:32:01.003921 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:01.003854 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:32:01.117886 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:01.117838 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-79b7c55f87-qm55w_353eb386-a09f-42e0-af32-c674bef8994a/cluster-samples-operator/0.log" Jul 22 00:32:01.315385 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:01.315359 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-79b7c55f87-qm55w_353eb386-a09f-42e0-af32-c674bef8994a/cluster-samples-operator-watch/0.log" Jul 22 00:32:03.167027 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:03.166991 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:32:03.239794 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:03.239747 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:32:06.095112 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.095069 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bc4c9b4c8-7tqc5" podUID="ac2409b4-9efc-4d67-af2c-6a67bf3b60c8" containerName="console" containerID="cri-o://4a55fbb2ea97fef10271d60e5dc2e85a6e29508cf1f917ec2ca3abaa8616b891" gracePeriod=15 Jul 22 00:32:06.219420 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.219392 2567 generic.go:356] "Generic (PLEG): container finished" podID="ac2409b4-9efc-4d67-af2c-6a67bf3b60c8" containerID="4a55fbb2ea97fef10271d60e5dc2e85a6e29508cf1f917ec2ca3abaa8616b891" exitCode=0 Jul 22 00:32:06.219525 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.219461 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc4c9b4c8-7tqc5" event={"ID":"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8","Type":"ContainerDied","Data":"4a55fbb2ea97fef10271d60e5dc2e85a6e29508cf1f917ec2ca3abaa8616b891"} Jul 22 00:32:06.376967 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.376944 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:32:06.450392 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.450369 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-service-ca\") pod \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " Jul 22 00:32:06.450529 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.450423 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-oauth-serving-cert\") pod \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " Jul 22 00:32:06.450529 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.450443 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-config\") pod \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " Jul 22 00:32:06.450529 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.450459 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svlk5\" (UniqueName: \"kubernetes.io/projected/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-kube-api-access-svlk5\") pod \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " Jul 22 00:32:06.450529 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.450491 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-oauth-config\") pod \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " Jul 22 00:32:06.450529 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.450514 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-serving-cert\") pod \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\" (UID: \"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8\") " Jul 22 00:32:06.450885 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.450853 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ac2409b4-9efc-4d67-af2c-6a67bf3b60c8" (UID: "ac2409b4-9efc-4d67-af2c-6a67bf3b60c8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:06.450885 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.450864 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-config" (OuterVolumeSpecName: "console-config") pod "ac2409b4-9efc-4d67-af2c-6a67bf3b60c8" (UID: "ac2409b4-9efc-4d67-af2c-6a67bf3b60c8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:06.450885 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.450850 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-service-ca" (OuterVolumeSpecName: "service-ca") pod "ac2409b4-9efc-4d67-af2c-6a67bf3b60c8" (UID: "ac2409b4-9efc-4d67-af2c-6a67bf3b60c8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:06.452710 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.452689 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-kube-api-access-svlk5" (OuterVolumeSpecName: "kube-api-access-svlk5") pod "ac2409b4-9efc-4d67-af2c-6a67bf3b60c8" (UID: "ac2409b4-9efc-4d67-af2c-6a67bf3b60c8"). InnerVolumeSpecName "kube-api-access-svlk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:32:06.452828 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.452729 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ac2409b4-9efc-4d67-af2c-6a67bf3b60c8" (UID: "ac2409b4-9efc-4d67-af2c-6a67bf3b60c8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:06.452828 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.452763 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ac2409b4-9efc-4d67-af2c-6a67bf3b60c8" (UID: "ac2409b4-9efc-4d67-af2c-6a67bf3b60c8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:06.551921 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.551898 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-oauth-serving-cert\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:06.551921 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.551919 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-config\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:06.552073 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.551928 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svlk5\" (UniqueName: \"kubernetes.io/projected/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-kube-api-access-svlk5\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:06.552073 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.551937 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-oauth-config\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:06.552073 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.551945 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-console-serving-cert\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:06.552073 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.551953 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8-service-ca\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:06.987261 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:06.987205 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" podUID="e0a1eb7a-b0dc-4804-975f-c601526d229e" containerName="registry" containerID="cri-o://48daa8950170c5d6a2e4aebf80e5bdab6b907cf3ddcf8fdb334afdac1e7c9eff" gracePeriod=30 Jul 22 00:32:07.224823 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.224795 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc4c9b4c8-7tqc5" Jul 22 00:32:07.224823 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.224815 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc4c9b4c8-7tqc5" event={"ID":"ac2409b4-9efc-4d67-af2c-6a67bf3b60c8","Type":"ContainerDied","Data":"d6ab2a330a34cd6b7210d92869d9403f0d1e4c69ace407fd9a177244edbc203a"} Jul 22 00:32:07.225250 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.224882 2567 scope.go:117] "RemoveContainer" containerID="4a55fbb2ea97fef10271d60e5dc2e85a6e29508cf1f917ec2ca3abaa8616b891" Jul 22 00:32:07.226760 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.226734 2567 generic.go:356] "Generic (PLEG): container finished" podID="e0a1eb7a-b0dc-4804-975f-c601526d229e" containerID="48daa8950170c5d6a2e4aebf80e5bdab6b907cf3ddcf8fdb334afdac1e7c9eff" exitCode=0 Jul 22 00:32:07.226873 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.226846 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" event={"ID":"e0a1eb7a-b0dc-4804-975f-c601526d229e","Type":"ContainerDied","Data":"48daa8950170c5d6a2e4aebf80e5bdab6b907cf3ddcf8fdb334afdac1e7c9eff"} Jul 22 00:32:07.252920 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.252864 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bc4c9b4c8-7tqc5"] Jul 22 00:32:07.254916 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.254897 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bc4c9b4c8-7tqc5"] Jul 22 00:32:07.272904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.272886 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:32:07.359959 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.359937 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls\") pod \"e0a1eb7a-b0dc-4804-975f-c601526d229e\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " Jul 22 00:32:07.360049 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.359973 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0a1eb7a-b0dc-4804-975f-c601526d229e-ca-trust-extracted\") pod \"e0a1eb7a-b0dc-4804-975f-c601526d229e\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " Jul 22 00:32:07.360049 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.360007 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-bound-sa-token\") pod \"e0a1eb7a-b0dc-4804-975f-c601526d229e\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " Jul 22 00:32:07.360049 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.360037 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0a1eb7a-b0dc-4804-975f-c601526d229e-installation-pull-secrets\") pod \"e0a1eb7a-b0dc-4804-975f-c601526d229e\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " Jul 22 00:32:07.360160 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.360063 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0a1eb7a-b0dc-4804-975f-c601526d229e-image-registry-private-configuration\") pod \"e0a1eb7a-b0dc-4804-975f-c601526d229e\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " Jul 22 00:32:07.360160 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.360111 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0a1eb7a-b0dc-4804-975f-c601526d229e-trusted-ca\") pod \"e0a1eb7a-b0dc-4804-975f-c601526d229e\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " Jul 22 00:32:07.360238 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.360167 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kqz4\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-kube-api-access-8kqz4\") pod \"e0a1eb7a-b0dc-4804-975f-c601526d229e\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " Jul 22 00:32:07.360238 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.360198 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-certificates\") pod \"e0a1eb7a-b0dc-4804-975f-c601526d229e\" (UID: \"e0a1eb7a-b0dc-4804-975f-c601526d229e\") " Jul 22 00:32:07.360793 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.360754 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a1eb7a-b0dc-4804-975f-c601526d229e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e0a1eb7a-b0dc-4804-975f-c601526d229e" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:07.360921 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.360896 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e0a1eb7a-b0dc-4804-975f-c601526d229e" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:07.362497 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.362462 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e0a1eb7a-b0dc-4804-975f-c601526d229e" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:32:07.362721 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.362682 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e0a1eb7a-b0dc-4804-975f-c601526d229e" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:32:07.362847 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.362767 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0a1eb7a-b0dc-4804-975f-c601526d229e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e0a1eb7a-b0dc-4804-975f-c601526d229e" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:07.362919 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.362865 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-kube-api-access-8kqz4" (OuterVolumeSpecName: "kube-api-access-8kqz4") pod "e0a1eb7a-b0dc-4804-975f-c601526d229e" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e"). InnerVolumeSpecName "kube-api-access-8kqz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:32:07.362919 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.362866 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0a1eb7a-b0dc-4804-975f-c601526d229e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e0a1eb7a-b0dc-4804-975f-c601526d229e" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:07.369767 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.369742 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a1eb7a-b0dc-4804-975f-c601526d229e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e0a1eb7a-b0dc-4804-975f-c601526d229e" (UID: "e0a1eb7a-b0dc-4804-975f-c601526d229e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:32:07.460961 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.460936 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kqz4\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-kube-api-access-8kqz4\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:07.460961 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.460958 2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-certificates\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:07.461078 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.460969 2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-registry-tls\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:07.461078 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.460977 2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0a1eb7a-b0dc-4804-975f-c601526d229e-ca-trust-extracted\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:07.461078 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.460985 2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0a1eb7a-b0dc-4804-975f-c601526d229e-bound-sa-token\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:07.461078 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.460994 2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0a1eb7a-b0dc-4804-975f-c601526d229e-installation-pull-secrets\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:07.461078 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.461002 2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0a1eb7a-b0dc-4804-975f-c601526d229e-image-registry-private-configuration\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:07.461078 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:07.461011 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0a1eb7a-b0dc-4804-975f-c601526d229e-trusted-ca\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:08.117752 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:08.117720 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac2409b4-9efc-4d67-af2c-6a67bf3b60c8" path="/var/lib/kubelet/pods/ac2409b4-9efc-4d67-af2c-6a67bf3b60c8/volumes" Jul 22 00:32:08.232452 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:08.232416 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" Jul 22 00:32:08.232452 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:08.232429 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58fcb4cbcd-j5fmf" event={"ID":"e0a1eb7a-b0dc-4804-975f-c601526d229e","Type":"ContainerDied","Data":"efb218e33ab871bcfb093d92184eb583b77e08be40f9108acace78225ad31f49"} Jul 22 00:32:08.233036 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:08.232488 2567 scope.go:117] "RemoveContainer" containerID="48daa8950170c5d6a2e4aebf80e5bdab6b907cf3ddcf8fdb334afdac1e7c9eff" Jul 22 00:32:08.248409 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:08.248388 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58fcb4cbcd-j5fmf"] Jul 22 00:32:08.253943 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:08.253922 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58fcb4cbcd-j5fmf"] Jul 22 00:32:10.117629 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:10.117597 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0a1eb7a-b0dc-4804-975f-c601526d229e" path="/var/lib/kubelet/pods/e0a1eb7a-b0dc-4804-975f-c601526d229e/volumes" Jul 22 00:32:13.432292 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:13.432249 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:32:13.488298 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:13.488257 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:32:24.297748 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:24.297713 2567 generic.go:356] "Generic (PLEG): container finished" podID="37f01742-c77e-4f48-8ab1-e9c4740f3d79" containerID="79a72fa55a8be87c5fee53814592474dd4c9cb444d5a42a2581935da6904d864" exitCode=0 Jul 22 00:32:24.298219 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:24.297805 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" event={"ID":"37f01742-c77e-4f48-8ab1-e9c4740f3d79","Type":"ContainerDied","Data":"79a72fa55a8be87c5fee53814592474dd4c9cb444d5a42a2581935da6904d864"} Jul 22 00:32:24.298219 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:24.298097 2567 scope.go:117] "RemoveContainer" containerID="79a72fa55a8be87c5fee53814592474dd4c9cb444d5a42a2581935da6904d864" Jul 22 00:32:25.303676 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:25.303597 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n9fxs" event={"ID":"37f01742-c77e-4f48-8ab1-e9c4740f3d79","Type":"ContainerStarted","Data":"4dc64854d61b67995b10d7fa5ce1442691eec203b8a6c7c8582c1ec9c233eaf4"} Jul 22 00:32:34.933580 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:34.933528 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:32:46.038506 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:46.038467 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:32:46.040960 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:46.040934 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d55898f6-dc0e-408a-99cc-9883ae591ac5-metrics-certs\") pod \"network-metrics-daemon-6q2tc\" (UID: \"d55898f6-dc0e-408a-99cc-9883ae591ac5\") " pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:32:46.318024 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:46.317950 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dh9z6\"" Jul 22 00:32:46.326018 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:46.326002 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6q2tc" Jul 22 00:32:46.453561 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:46.453514 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6q2tc"] Jul 22 00:32:46.455649 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:32:46.455622 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55898f6_dc0e_408a_99cc_9883ae591ac5.slice/crio-90cfa8def8f95d93783e6c79109fb7616d9eab60b1cde8c5d630d0338f6db320 WatchSource:0}: Error finding container 90cfa8def8f95d93783e6c79109fb7616d9eab60b1cde8c5d630d0338f6db320: Status 404 returned error can't find the container with id 90cfa8def8f95d93783e6c79109fb7616d9eab60b1cde8c5d630d0338f6db320 Jul 22 00:32:47.387125 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:47.387092 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6q2tc" event={"ID":"d55898f6-dc0e-408a-99cc-9883ae591ac5","Type":"ContainerStarted","Data":"90cfa8def8f95d93783e6c79109fb7616d9eab60b1cde8c5d630d0338f6db320"} Jul 22 00:32:48.392876 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:48.392839 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6q2tc" event={"ID":"d55898f6-dc0e-408a-99cc-9883ae591ac5","Type":"ContainerStarted","Data":"f5d0440fb66c830594eb461f372a5b3cd800d01446a108ffac175080d4ee6fc9"} Jul 22 00:32:48.393251 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:48.392901 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6q2tc" event={"ID":"d55898f6-dc0e-408a-99cc-9883ae591ac5","Type":"ContainerStarted","Data":"91fbea25d879085b5dca5d1894eedd6dbadfbb2ea920ba3b53d0ebf2c937266d"} Jul 22 00:32:48.409409 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:48.409361 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6q2tc" podStartSLOduration=375.451450096 podStartE2EDuration="6m16.409347394s" totalImagesPullingTime="957.897298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:32:48.407946006 +0000 UTC m=+376.734018208" watchObservedRunningTime="2026-07-22 00:32:48.409347394 +0000 UTC m=+376.735419622" Jul 22 00:32:48.497409 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:48.497371 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:32:48.598244 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:48.598218 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:32:53.114342 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:53.114304 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:32:53.326402 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:53.326358 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:32:54.125032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.124999 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 00:32:54.125417 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.125295 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="alertmanager" containerID="cri-o://b8fadde9c9aa8a92a1949afe1522c1c364e66c9e17d84f9c18ded3e8cdb3777b" gracePeriod=120 Jul 22 00:32:54.125417 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.125350 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="kube-rbac-proxy-metric" containerID="cri-o://c98571b4834b3152d4991c4bd843c576703908eb711cb01758bb18cd74fdeebb" gracePeriod=120 Jul 22 00:32:54.125530 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.125389 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="kube-rbac-proxy-web" containerID="cri-o://8735173f284503ea25529ae96d2257f8cb3ca625fba9afa4eb6a69ee8ce94b03" gracePeriod=120 Jul 22 00:32:54.125530 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.125451 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="kube-rbac-proxy" containerID="cri-o://aa6d7e10a676e4ff68d39d4171e2f65a5b428a20479c6512574975cb38ab1302" gracePeriod=120 Jul 22 00:32:54.125530 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.125446 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="config-reloader" containerID="cri-o://7513a18cd70599e687216678e3569b5313b648ad69020b1ea582cd753fbf0693" gracePeriod=120 Jul 22 00:32:54.125530 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.125408 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="prom-label-proxy" containerID="cri-o://b0d8c1b3e651385ae6a2aede999e79c839aef4514be75bd7d73f27694f0b8eeb" gracePeriod=120 Jul 22 00:32:54.421407 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.421335 2567 generic.go:356] "Generic (PLEG): container finished" podID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerID="b0d8c1b3e651385ae6a2aede999e79c839aef4514be75bd7d73f27694f0b8eeb" exitCode=0 Jul 22 00:32:54.421407 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.421365 2567 generic.go:356] "Generic (PLEG): container finished" podID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerID="c98571b4834b3152d4991c4bd843c576703908eb711cb01758bb18cd74fdeebb" exitCode=0 Jul 22 00:32:54.421407 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.421372 2567 generic.go:356] "Generic (PLEG): container finished" podID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerID="aa6d7e10a676e4ff68d39d4171e2f65a5b428a20479c6512574975cb38ab1302" exitCode=0 Jul 22 00:32:54.421407 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.421378 2567 generic.go:356] "Generic (PLEG): container finished" podID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerID="7513a18cd70599e687216678e3569b5313b648ad69020b1ea582cd753fbf0693" exitCode=0 Jul 22 00:32:54.421407 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.421383 2567 generic.go:356] "Generic (PLEG): container finished" podID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerID="b8fadde9c9aa8a92a1949afe1522c1c364e66c9e17d84f9c18ded3e8cdb3777b" exitCode=0 Jul 22 00:32:54.421719 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.421411 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cef264aa-037a-482c-ba8e-dbca38f79a27","Type":"ContainerDied","Data":"b0d8c1b3e651385ae6a2aede999e79c839aef4514be75bd7d73f27694f0b8eeb"} Jul 22 00:32:54.421719 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.421502 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cef264aa-037a-482c-ba8e-dbca38f79a27","Type":"ContainerDied","Data":"c98571b4834b3152d4991c4bd843c576703908eb711cb01758bb18cd74fdeebb"} Jul 22 00:32:54.421719 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.421544 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cef264aa-037a-482c-ba8e-dbca38f79a27","Type":"ContainerDied","Data":"aa6d7e10a676e4ff68d39d4171e2f65a5b428a20479c6512574975cb38ab1302"} Jul 22 00:32:54.421719 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.421589 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cef264aa-037a-482c-ba8e-dbca38f79a27","Type":"ContainerDied","Data":"7513a18cd70599e687216678e3569b5313b648ad69020b1ea582cd753fbf0693"} Jul 22 00:32:54.421719 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:54.421632 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cef264aa-037a-482c-ba8e-dbca38f79a27","Type":"ContainerDied","Data":"b8fadde9c9aa8a92a1949afe1522c1c364e66c9e17d84f9c18ded3e8cdb3777b"} Jul 22 00:32:55.370712 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.370688 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.405633 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.405603 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy\") pod \"cef264aa-037a-482c-ba8e-dbca38f79a27\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " Jul 22 00:32:55.405989 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.405964 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy-metric\") pod \"cef264aa-037a-482c-ba8e-dbca38f79a27\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " Jul 22 00:32:55.406072 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.406011 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cef264aa-037a-482c-ba8e-dbca38f79a27-config-out\") pod \"cef264aa-037a-482c-ba8e-dbca38f79a27\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " Jul 22 00:32:55.406134 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.406075 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cef264aa-037a-482c-ba8e-dbca38f79a27-alertmanager-trusted-ca-bundle\") pod \"cef264aa-037a-482c-ba8e-dbca38f79a27\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " Jul 22 00:32:55.406134 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.406105 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cef264aa-037a-482c-ba8e-dbca38f79a27-tls-assets\") pod \"cef264aa-037a-482c-ba8e-dbca38f79a27\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " Jul 22 00:32:55.406234 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.406139 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-web-config\") pod \"cef264aa-037a-482c-ba8e-dbca38f79a27\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " Jul 22 00:32:55.406234 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.406164 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cef264aa-037a-482c-ba8e-dbca38f79a27-alertmanager-main-db\") pod \"cef264aa-037a-482c-ba8e-dbca38f79a27\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " Jul 22 00:32:55.406234 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.406193 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cef264aa-037a-482c-ba8e-dbca38f79a27-metrics-client-ca\") pod \"cef264aa-037a-482c-ba8e-dbca38f79a27\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " Jul 22 00:32:55.406234 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.406226 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-cluster-tls-config\") pod \"cef264aa-037a-482c-ba8e-dbca38f79a27\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " Jul 22 00:32:55.406424 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.406253 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jphxm\" (UniqueName: \"kubernetes.io/projected/cef264aa-037a-482c-ba8e-dbca38f79a27-kube-api-access-jphxm\") pod \"cef264aa-037a-482c-ba8e-dbca38f79a27\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " Jul 22 00:32:55.406424 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.406286 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-main-tls\") pod \"cef264aa-037a-482c-ba8e-dbca38f79a27\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " Jul 22 00:32:55.406424 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.406334 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-config-volume\") pod \"cef264aa-037a-482c-ba8e-dbca38f79a27\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " Jul 22 00:32:55.406424 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.406370 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy-web\") pod \"cef264aa-037a-482c-ba8e-dbca38f79a27\" (UID: \"cef264aa-037a-482c-ba8e-dbca38f79a27\") " Jul 22 00:32:55.407388 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.407201 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef264aa-037a-482c-ba8e-dbca38f79a27-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "cef264aa-037a-482c-ba8e-dbca38f79a27" (UID: "cef264aa-037a-482c-ba8e-dbca38f79a27"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:32:55.407388 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.407349 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef264aa-037a-482c-ba8e-dbca38f79a27-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "cef264aa-037a-482c-ba8e-dbca38f79a27" (UID: "cef264aa-037a-482c-ba8e-dbca38f79a27"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:55.407552 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.407529 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef264aa-037a-482c-ba8e-dbca38f79a27-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "cef264aa-037a-482c-ba8e-dbca38f79a27" (UID: "cef264aa-037a-482c-ba8e-dbca38f79a27"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:55.410019 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.409948 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "cef264aa-037a-482c-ba8e-dbca38f79a27" (UID: "cef264aa-037a-482c-ba8e-dbca38f79a27"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:55.410626 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.410590 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "cef264aa-037a-482c-ba8e-dbca38f79a27" (UID: "cef264aa-037a-482c-ba8e-dbca38f79a27"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:55.412654 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.412629 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-config-volume" (OuterVolumeSpecName: "config-volume") pod "cef264aa-037a-482c-ba8e-dbca38f79a27" (UID: "cef264aa-037a-482c-ba8e-dbca38f79a27"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:55.412752 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.412649 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef264aa-037a-482c-ba8e-dbca38f79a27-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "cef264aa-037a-482c-ba8e-dbca38f79a27" (UID: "cef264aa-037a-482c-ba8e-dbca38f79a27"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:32:55.412752 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.412744 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "cef264aa-037a-482c-ba8e-dbca38f79a27" (UID: "cef264aa-037a-482c-ba8e-dbca38f79a27"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:55.413007 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.412971 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "cef264aa-037a-482c-ba8e-dbca38f79a27" (UID: "cef264aa-037a-482c-ba8e-dbca38f79a27"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:55.413141 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.413046 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef264aa-037a-482c-ba8e-dbca38f79a27-config-out" (OuterVolumeSpecName: "config-out") pod "cef264aa-037a-482c-ba8e-dbca38f79a27" (UID: "cef264aa-037a-482c-ba8e-dbca38f79a27"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:32:55.413207 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.413184 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef264aa-037a-482c-ba8e-dbca38f79a27-kube-api-access-jphxm" (OuterVolumeSpecName: "kube-api-access-jphxm") pod "cef264aa-037a-482c-ba8e-dbca38f79a27" (UID: "cef264aa-037a-482c-ba8e-dbca38f79a27"). InnerVolumeSpecName "kube-api-access-jphxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:32:55.416885 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.416858 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "cef264aa-037a-482c-ba8e-dbca38f79a27" (UID: "cef264aa-037a-482c-ba8e-dbca38f79a27"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:55.425667 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.425643 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-web-config" (OuterVolumeSpecName: "web-config") pod "cef264aa-037a-482c-ba8e-dbca38f79a27" (UID: "cef264aa-037a-482c-ba8e-dbca38f79a27"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:55.428733 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.428710 2567 generic.go:356] "Generic (PLEG): container finished" podID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerID="8735173f284503ea25529ae96d2257f8cb3ca625fba9afa4eb6a69ee8ce94b03" exitCode=0 Jul 22 00:32:55.428871 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.428797 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cef264aa-037a-482c-ba8e-dbca38f79a27","Type":"ContainerDied","Data":"8735173f284503ea25529ae96d2257f8cb3ca625fba9afa4eb6a69ee8ce94b03"} Jul 22 00:32:55.428871 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.428827 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.428871 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.428867 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cef264aa-037a-482c-ba8e-dbca38f79a27","Type":"ContainerDied","Data":"b03e265da2b977b8a88134a85b4fe2e2131ff224b9de73765db733049e432adc"} Jul 22 00:32:55.428999 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.428905 2567 scope.go:117] "RemoveContainer" containerID="b0d8c1b3e651385ae6a2aede999e79c839aef4514be75bd7d73f27694f0b8eeb" Jul 22 00:32:55.437222 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.437207 2567 scope.go:117] "RemoveContainer" containerID="c98571b4834b3152d4991c4bd843c576703908eb711cb01758bb18cd74fdeebb" Jul 22 00:32:55.444954 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.444935 2567 scope.go:117] "RemoveContainer" containerID="aa6d7e10a676e4ff68d39d4171e2f65a5b428a20479c6512574975cb38ab1302" Jul 22 00:32:55.451452 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.451430 2567 scope.go:117] "RemoveContainer" containerID="8735173f284503ea25529ae96d2257f8cb3ca625fba9afa4eb6a69ee8ce94b03" Jul 22 00:32:55.453557 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.453535 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 00:32:55.458626 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.458553 2567 scope.go:117] "RemoveContainer" containerID="7513a18cd70599e687216678e3569b5313b648ad69020b1ea582cd753fbf0693" Jul 22 00:32:55.459767 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.459746 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 00:32:55.465316 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.465296 2567 scope.go:117] "RemoveContainer" containerID="b8fadde9c9aa8a92a1949afe1522c1c364e66c9e17d84f9c18ded3e8cdb3777b" Jul 22 00:32:55.471823 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.471805 2567 scope.go:117] "RemoveContainer" containerID="42c35dc186e02e76fccaa16d01116fb477681cffb70c81be5aeba5ccf6f92783" Jul 22 00:32:55.478032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.478016 2567 scope.go:117] "RemoveContainer" containerID="b0d8c1b3e651385ae6a2aede999e79c839aef4514be75bd7d73f27694f0b8eeb" Jul 22 00:32:55.478343 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:32:55.478314 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d8c1b3e651385ae6a2aede999e79c839aef4514be75bd7d73f27694f0b8eeb\": container with ID starting with b0d8c1b3e651385ae6a2aede999e79c839aef4514be75bd7d73f27694f0b8eeb not found: ID does not exist" containerID="b0d8c1b3e651385ae6a2aede999e79c839aef4514be75bd7d73f27694f0b8eeb" Jul 22 00:32:55.478407 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.478351 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d8c1b3e651385ae6a2aede999e79c839aef4514be75bd7d73f27694f0b8eeb"} err="failed to get container status \"b0d8c1b3e651385ae6a2aede999e79c839aef4514be75bd7d73f27694f0b8eeb\": rpc error: code = NotFound desc = could not find container \"b0d8c1b3e651385ae6a2aede999e79c839aef4514be75bd7d73f27694f0b8eeb\": container with ID starting with b0d8c1b3e651385ae6a2aede999e79c839aef4514be75bd7d73f27694f0b8eeb not found: ID does not exist" Jul 22 00:32:55.478407 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.478381 2567 scope.go:117] "RemoveContainer" containerID="c98571b4834b3152d4991c4bd843c576703908eb711cb01758bb18cd74fdeebb" Jul 22 00:32:55.478630 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:32:55.478612 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98571b4834b3152d4991c4bd843c576703908eb711cb01758bb18cd74fdeebb\": container with ID starting with c98571b4834b3152d4991c4bd843c576703908eb711cb01758bb18cd74fdeebb not found: ID does not exist" containerID="c98571b4834b3152d4991c4bd843c576703908eb711cb01758bb18cd74fdeebb" Jul 22 00:32:55.478673 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.478635 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98571b4834b3152d4991c4bd843c576703908eb711cb01758bb18cd74fdeebb"} err="failed to get container status \"c98571b4834b3152d4991c4bd843c576703908eb711cb01758bb18cd74fdeebb\": rpc error: code = NotFound desc = could not find container \"c98571b4834b3152d4991c4bd843c576703908eb711cb01758bb18cd74fdeebb\": container with ID starting with c98571b4834b3152d4991c4bd843c576703908eb711cb01758bb18cd74fdeebb not found: ID does not exist" Jul 22 00:32:55.478673 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.478650 2567 scope.go:117] "RemoveContainer" containerID="aa6d7e10a676e4ff68d39d4171e2f65a5b428a20479c6512574975cb38ab1302" Jul 22 00:32:55.478875 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:32:55.478859 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa6d7e10a676e4ff68d39d4171e2f65a5b428a20479c6512574975cb38ab1302\": container with ID starting with aa6d7e10a676e4ff68d39d4171e2f65a5b428a20479c6512574975cb38ab1302 not found: ID does not exist" containerID="aa6d7e10a676e4ff68d39d4171e2f65a5b428a20479c6512574975cb38ab1302" Jul 22 00:32:55.478919 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.478882 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa6d7e10a676e4ff68d39d4171e2f65a5b428a20479c6512574975cb38ab1302"} err="failed to get container status \"aa6d7e10a676e4ff68d39d4171e2f65a5b428a20479c6512574975cb38ab1302\": rpc error: code = NotFound desc = could not find container \"aa6d7e10a676e4ff68d39d4171e2f65a5b428a20479c6512574975cb38ab1302\": container with ID starting with aa6d7e10a676e4ff68d39d4171e2f65a5b428a20479c6512574975cb38ab1302 not found: ID does not exist" Jul 22 00:32:55.478919 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.478894 2567 scope.go:117] "RemoveContainer" containerID="8735173f284503ea25529ae96d2257f8cb3ca625fba9afa4eb6a69ee8ce94b03" Jul 22 00:32:55.479129 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:32:55.479112 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8735173f284503ea25529ae96d2257f8cb3ca625fba9afa4eb6a69ee8ce94b03\": container with ID starting with 8735173f284503ea25529ae96d2257f8cb3ca625fba9afa4eb6a69ee8ce94b03 not found: ID does not exist" containerID="8735173f284503ea25529ae96d2257f8cb3ca625fba9afa4eb6a69ee8ce94b03" Jul 22 00:32:55.479164 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.479134 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8735173f284503ea25529ae96d2257f8cb3ca625fba9afa4eb6a69ee8ce94b03"} err="failed to get container status \"8735173f284503ea25529ae96d2257f8cb3ca625fba9afa4eb6a69ee8ce94b03\": rpc error: code = NotFound desc = could not find container \"8735173f284503ea25529ae96d2257f8cb3ca625fba9afa4eb6a69ee8ce94b03\": container with ID starting with 8735173f284503ea25529ae96d2257f8cb3ca625fba9afa4eb6a69ee8ce94b03 not found: ID does not exist" Jul 22 00:32:55.479164 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.479149 2567 scope.go:117] "RemoveContainer" containerID="7513a18cd70599e687216678e3569b5313b648ad69020b1ea582cd753fbf0693" Jul 22 00:32:55.479360 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:32:55.479345 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7513a18cd70599e687216678e3569b5313b648ad69020b1ea582cd753fbf0693\": container with ID starting with 7513a18cd70599e687216678e3569b5313b648ad69020b1ea582cd753fbf0693 not found: ID does not exist" containerID="7513a18cd70599e687216678e3569b5313b648ad69020b1ea582cd753fbf0693" Jul 22 00:32:55.479401 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.479366 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7513a18cd70599e687216678e3569b5313b648ad69020b1ea582cd753fbf0693"} err="failed to get container status \"7513a18cd70599e687216678e3569b5313b648ad69020b1ea582cd753fbf0693\": rpc error: code = NotFound desc = could not find container \"7513a18cd70599e687216678e3569b5313b648ad69020b1ea582cd753fbf0693\": container with ID starting with 7513a18cd70599e687216678e3569b5313b648ad69020b1ea582cd753fbf0693 not found: ID does not exist" Jul 22 00:32:55.479401 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.479379 2567 scope.go:117] "RemoveContainer" containerID="b8fadde9c9aa8a92a1949afe1522c1c364e66c9e17d84f9c18ded3e8cdb3777b" Jul 22 00:32:55.479577 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:32:55.479561 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8fadde9c9aa8a92a1949afe1522c1c364e66c9e17d84f9c18ded3e8cdb3777b\": container with ID starting with b8fadde9c9aa8a92a1949afe1522c1c364e66c9e17d84f9c18ded3e8cdb3777b not found: ID does not exist" containerID="b8fadde9c9aa8a92a1949afe1522c1c364e66c9e17d84f9c18ded3e8cdb3777b" Jul 22 00:32:55.479622 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.479580 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8fadde9c9aa8a92a1949afe1522c1c364e66c9e17d84f9c18ded3e8cdb3777b"} err="failed to get container status \"b8fadde9c9aa8a92a1949afe1522c1c364e66c9e17d84f9c18ded3e8cdb3777b\": rpc error: code = NotFound desc = could not find container \"b8fadde9c9aa8a92a1949afe1522c1c364e66c9e17d84f9c18ded3e8cdb3777b\": container with ID starting with b8fadde9c9aa8a92a1949afe1522c1c364e66c9e17d84f9c18ded3e8cdb3777b not found: ID does not exist" Jul 22 00:32:55.479622 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.479593 2567 scope.go:117] "RemoveContainer" containerID="42c35dc186e02e76fccaa16d01116fb477681cffb70c81be5aeba5ccf6f92783" Jul 22 00:32:55.479911 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:32:55.479888 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c35dc186e02e76fccaa16d01116fb477681cffb70c81be5aeba5ccf6f92783\": container with ID starting with 42c35dc186e02e76fccaa16d01116fb477681cffb70c81be5aeba5ccf6f92783 not found: ID does not exist" containerID="42c35dc186e02e76fccaa16d01116fb477681cffb70c81be5aeba5ccf6f92783" Jul 22 00:32:55.479959 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.479918 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c35dc186e02e76fccaa16d01116fb477681cffb70c81be5aeba5ccf6f92783"} err="failed to get container status \"42c35dc186e02e76fccaa16d01116fb477681cffb70c81be5aeba5ccf6f92783\": rpc error: code = NotFound desc = could not find container \"42c35dc186e02e76fccaa16d01116fb477681cffb70c81be5aeba5ccf6f92783\": container with ID starting with 42c35dc186e02e76fccaa16d01116fb477681cffb70c81be5aeba5ccf6f92783 not found: ID does not exist" Jul 22 00:32:55.486677 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.486659 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 00:32:55.486952 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.486938 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="kube-rbac-proxy-metric" Jul 22 00:32:55.486996 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.486954 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="kube-rbac-proxy-metric" Jul 22 00:32:55.486996 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.486967 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac2409b4-9efc-4d67-af2c-6a67bf3b60c8" containerName="console" Jul 22 00:32:55.486996 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.486973 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac2409b4-9efc-4d67-af2c-6a67bf3b60c8" containerName="console" Jul 22 00:32:55.486996 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.486981 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="config-reloader" Jul 22 00:32:55.486996 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.486986 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="config-reloader" Jul 22 00:32:55.486996 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.486993 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="kube-rbac-proxy" Jul 22 00:32:55.486996 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.486998 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="kube-rbac-proxy" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487009 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="init-config-reloader" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487014 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="init-config-reloader" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487021 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a1eb7a-b0dc-4804-975f-c601526d229e" containerName="registry" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487026 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a1eb7a-b0dc-4804-975f-c601526d229e" containerName="registry" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487033 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="alertmanager" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487037 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="alertmanager" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487042 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="prom-label-proxy" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487048 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="prom-label-proxy" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487054 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="kube-rbac-proxy-web" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487059 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="kube-rbac-proxy-web" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487098 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a1eb7a-b0dc-4804-975f-c601526d229e" containerName="registry" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487108 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac2409b4-9efc-4d67-af2c-6a67bf3b60c8" containerName="console" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487114 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="alertmanager" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487120 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="kube-rbac-proxy-web" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487125 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="config-reloader" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487130 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="kube-rbac-proxy" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487135 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="kube-rbac-proxy-metric" Jul 22 00:32:55.487199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.487140 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" containerName="prom-label-proxy" Jul 22 00:32:55.491907 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.491889 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.494227 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.494208 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 00:32:55.494324 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.494246 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 00:32:55.494324 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.494256 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 00:32:55.494416 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.494323 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 00:32:55.494517 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.494504 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 00:32:55.494651 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.494641 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 00:32:55.494732 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.494718 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 00:32:55.494954 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.494938 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 00:32:55.495157 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.495141 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zsft8\"" Jul 22 00:32:55.501046 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.501026 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 00:32:55.506939 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.506916 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80e5b98b-dcff-45b9-ac36-c3007953b774-tls-assets\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.507045 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.506964 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e5b98b-dcff-45b9-ac36-c3007953b774-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.507045 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.506991 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.507045 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507016 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80e5b98b-dcff-45b9-ac36-c3007953b774-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.507205 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507046 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-web-config\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.507205 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507065 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztbwg\" (UniqueName: \"kubernetes.io/projected/80e5b98b-dcff-45b9-ac36-c3007953b774-kube-api-access-ztbwg\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.507205 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507081 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.507205 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507111 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-config-volume\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.507205 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507129 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.507205 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507156 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.507205 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507190 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/80e5b98b-dcff-45b9-ac36-c3007953b774-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.507516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507223 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.507516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507240 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80e5b98b-dcff-45b9-ac36-c3007953b774-config-out\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.507516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507280 2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:55.507516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507290 2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cef264aa-037a-482c-ba8e-dbca38f79a27-config-out\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:55.507516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507299 2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cef264aa-037a-482c-ba8e-dbca38f79a27-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:55.507516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507307 2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cef264aa-037a-482c-ba8e-dbca38f79a27-tls-assets\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:55.507516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507316 2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-web-config\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:55.507516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507323 2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cef264aa-037a-482c-ba8e-dbca38f79a27-alertmanager-main-db\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:55.507516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507332 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cef264aa-037a-482c-ba8e-dbca38f79a27-metrics-client-ca\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:55.507516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507340 2567 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-cluster-tls-config\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:55.507516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507347 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jphxm\" (UniqueName: \"kubernetes.io/projected/cef264aa-037a-482c-ba8e-dbca38f79a27-kube-api-access-jphxm\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:55.507516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507356 2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-main-tls\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:55.507516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507366 2567 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-config-volume\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:55.507516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507374 2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:55.507516 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.507383 2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cef264aa-037a-482c-ba8e-dbca38f79a27-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:32:55.508284 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.508176 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 00:32:55.608360 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.608330 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80e5b98b-dcff-45b9-ac36-c3007953b774-config-out\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.608475 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.608375 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80e5b98b-dcff-45b9-ac36-c3007953b774-tls-assets\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.608475 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.608397 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e5b98b-dcff-45b9-ac36-c3007953b774-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.608475 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.608415 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.608475 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.608432 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80e5b98b-dcff-45b9-ac36-c3007953b774-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.608475 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.608451 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-web-config\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.608891 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.608856 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztbwg\" (UniqueName: \"kubernetes.io/projected/80e5b98b-dcff-45b9-ac36-c3007953b774-kube-api-access-ztbwg\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.609001 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.608924 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.609286 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.609264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-config-volume\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.609365 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.609301 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80e5b98b-dcff-45b9-ac36-c3007953b774-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.609365 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.609312 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.609365 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.609359 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.609521 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.609387 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/80e5b98b-dcff-45b9-ac36-c3007953b774-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.609521 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.609399 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e5b98b-dcff-45b9-ac36-c3007953b774-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.609521 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.609428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.611239 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.611213 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80e5b98b-dcff-45b9-ac36-c3007953b774-config-out\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.611464 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.611384 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.611464 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.611397 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80e5b98b-dcff-45b9-ac36-c3007953b774-tls-assets\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.611628 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.611519 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/80e5b98b-dcff-45b9-ac36-c3007953b774-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.611628 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.611547 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-web-config\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.611851 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.611828 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.612032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.612010 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.612129 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.612109 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.612304 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.612286 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.613374 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.613354 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/80e5b98b-dcff-45b9-ac36-c3007953b774-config-volume\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.616380 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.616362 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztbwg\" (UniqueName: \"kubernetes.io/projected/80e5b98b-dcff-45b9-ac36-c3007953b774-kube-api-access-ztbwg\") pod \"alertmanager-main-0\" (UID: \"80e5b98b-dcff-45b9-ac36-c3007953b774\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.801479 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.801455 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 00:32:55.936055 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:55.935955 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 00:32:55.938573 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:32:55.938544 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e5b98b_dcff_45b9_ac36_c3007953b774.slice/crio-310c1f2e25496dbf38610bc9d173ab06187e4a14a87d8591de53e316fc6d7c85 WatchSource:0}: Error finding container 310c1f2e25496dbf38610bc9d173ab06187e4a14a87d8591de53e316fc6d7c85: Status 404 returned error can't find the container with id 310c1f2e25496dbf38610bc9d173ab06187e4a14a87d8591de53e316fc6d7c85 Jul 22 00:32:56.117204 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:56.117124 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cef264aa-037a-482c-ba8e-dbca38f79a27" path="/var/lib/kubelet/pods/cef264aa-037a-482c-ba8e-dbca38f79a27/volumes" Jul 22 00:32:56.436322 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:56.436245 2567 generic.go:356] "Generic (PLEG): container finished" podID="80e5b98b-dcff-45b9-ac36-c3007953b774" containerID="e9988f90df810b7d5aab58a277b3401fc834fe890de22f90c72961addc97ae78" exitCode=0 Jul 22 00:32:56.436322 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:56.436299 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80e5b98b-dcff-45b9-ac36-c3007953b774","Type":"ContainerDied","Data":"e9988f90df810b7d5aab58a277b3401fc834fe890de22f90c72961addc97ae78"} Jul 22 00:32:56.436697 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:56.436362 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80e5b98b-dcff-45b9-ac36-c3007953b774","Type":"ContainerStarted","Data":"310c1f2e25496dbf38610bc9d173ab06187e4a14a87d8591de53e316fc6d7c85"} Jul 22 00:32:57.443879 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:57.443846 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80e5b98b-dcff-45b9-ac36-c3007953b774","Type":"ContainerStarted","Data":"f243bd22e5522e297ec1d8f26f2c4bd2c45c451b19b2c81564358e37d77d8b65"} Jul 22 00:32:57.444216 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:57.443911 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80e5b98b-dcff-45b9-ac36-c3007953b774","Type":"ContainerStarted","Data":"00eaec161ba97989f635b7cc686bc185266d1961e64d30cc161e53950333d24b"} Jul 22 00:32:57.444216 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:57.443937 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80e5b98b-dcff-45b9-ac36-c3007953b774","Type":"ContainerStarted","Data":"c40eb81cfa8b17f6b002492b4bc0a3c5b9d92282962f9a70174ddaaab35ae19b"} Jul 22 00:32:57.444216 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:57.443962 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80e5b98b-dcff-45b9-ac36-c3007953b774","Type":"ContainerStarted","Data":"7abad440728297f03b14c5f5c857ac2e0c577fc950ab8b119e5f7d0e001d2cb2"} Jul 22 00:32:57.444216 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:57.443996 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80e5b98b-dcff-45b9-ac36-c3007953b774","Type":"ContainerStarted","Data":"395d32cc074df180f9499074cdfa4e7b2f8d42e82166566e1cf0cf57ae25480a"} Jul 22 00:32:57.444216 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:57.444024 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80e5b98b-dcff-45b9-ac36-c3007953b774","Type":"ContainerStarted","Data":"5fe41b33bf559e3dcef42a43b2cbe9e3682f309d1d6570e9ec22e4b7e67a921b"} Jul 22 00:32:57.471224 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:57.471161 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.456330145 podStartE2EDuration="2.471146083s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.815938ms" isStatefulPod=false podCreationTimestamp="2026-07-22 00:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:32:57.46858433 +0000 UTC m=+385.794656531" watchObservedRunningTime="2026-07-22 00:32:57.471146083 +0000 UTC m=+385.797218285" Jul 22 00:32:58.212104 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.212074 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-79bddbf7-dgpjb"] Jul 22 00:32:58.215374 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.215356 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.217712 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.217690 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 22 00:32:58.217937 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.217918 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 22 00:32:58.218043 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.217988 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 22 00:32:58.218043 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.218023 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qd8rs\"" Jul 22 00:32:58.218159 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.218114 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 22 00:32:58.218254 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.218233 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 22 00:32:58.222736 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.222713 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 22 00:32:58.225925 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.225904 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79bddbf7-dgpjb"] Jul 22 00:32:58.231344 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.231324 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-federate-client-tls\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.231438 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.231355 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-secret-telemeter-client\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.231438 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.231372 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-telemeter-client-tls\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.231558 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.231434 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.231558 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.231493 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-metrics-client-ca\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.231558 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.231540 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jmrh\" (UniqueName: \"kubernetes.io/projected/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-kube-api-access-6jmrh\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.231691 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.231585 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-serving-certs-ca-bundle\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.231691 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.231610 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.332294 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.332269 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-serving-certs-ca-bundle\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.332389 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.332297 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.332389 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.332320 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-federate-client-tls\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.332389 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.332341 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-secret-telemeter-client\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.332389 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.332365 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-telemeter-client-tls\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.332595 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.332390 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.332595 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.332417 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-metrics-client-ca\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.332595 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.332452 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmrh\" (UniqueName: \"kubernetes.io/projected/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-kube-api-access-6jmrh\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.333113 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.333087 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-serving-certs-ca-bundle\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.333256 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.333231 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.333354 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.333231 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-metrics-client-ca\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.334996 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.334976 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-federate-client-tls\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.335103 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.335064 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.335378 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.335353 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-telemeter-client-tls\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.335421 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.335361 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-secret-telemeter-client\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.344681 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.344661 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmrh\" (UniqueName: \"kubernetes.io/projected/4ba8c1a9-42f9-4d53-8943-88f0b2a97e34-kube-api-access-6jmrh\") pod \"telemeter-client-79bddbf7-dgpjb\" (UID: \"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34\") " pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.526026 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.526001 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" Jul 22 00:32:58.653161 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:58.653123 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79bddbf7-dgpjb"] Jul 22 00:32:58.655204 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:32:58.655183 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba8c1a9_42f9_4d53_8943_88f0b2a97e34.slice/crio-b1e29598544839cf38d7ef38fb6cf949315084d518f5746a4d5fac9318dad9e8 WatchSource:0}: Error finding container b1e29598544839cf38d7ef38fb6cf949315084d518f5746a4d5fac9318dad9e8: Status 404 returned error can't find the container with id b1e29598544839cf38d7ef38fb6cf949315084d518f5746a4d5fac9318dad9e8 Jul 22 00:32:59.455980 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:32:59.455946 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" event={"ID":"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34","Type":"ContainerStarted","Data":"b1e29598544839cf38d7ef38fb6cf949315084d518f5746a4d5fac9318dad9e8"} Jul 22 00:33:01.467241 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:33:01.467200 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" event={"ID":"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34","Type":"ContainerStarted","Data":"70d16d904ac91d1f73e5a4908208f1714b7773156f66e85c1db642c96551f649"} Jul 22 00:33:01.467613 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:33:01.467265 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" event={"ID":"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34","Type":"ContainerStarted","Data":"08a81df7b366e7fcfa4d04555370b562c64718ee2f472151ca2317ac6e730167"} Jul 22 00:33:01.467613 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:33:01.467286 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" event={"ID":"4ba8c1a9-42f9-4d53-8943-88f0b2a97e34","Type":"ContainerStarted","Data":"b64ceb931e2613b448540f5aaddc22fba46a8a3f687e0c24a0b1e49d64077f70"} Jul 22 00:33:01.490159 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:33:01.490113 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-79bddbf7-dgpjb" podStartSLOduration=1.317614111 podStartE2EDuration="3.49009655s" totalImagesPullingTime="2.172482439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:33:01.489117636 +0000 UTC m=+389.815189902" watchObservedRunningTime="2026-07-22 00:33:01.49009655 +0000 UTC m=+389.816168752" Jul 22 00:33:02.259201 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:33:02.259156 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:33:02.312734 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:33:02.312704 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:33:11.447225 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:33:11.447194 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:33:11.509871 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:33:11.509846 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:33:12.285642 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:33:12.285607 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:33:12.338336 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:33:12.338307 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:33:13.504955 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:33:13.504916 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:33:13.605640 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:33:13.605609 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:33:13.714128 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:33:13.714095 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:33:30.077748 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:33:30.077700 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:33:30.128462 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:33:30.128433 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:34:08.305532 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:08.305495 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:34:08.360559 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:08.360528 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:34:10.989089 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:10.989049 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bcc876b87-9kcfx"] Jul 22 00:34:10.994713 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:10.994667 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:10.998789 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:10.998736 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 00:34:10.998957 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:10.998915 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 00:34:10.999665 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:10.999630 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 00:34:10.999909 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:10.999797 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 00:34:10.999909 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:10.999816 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 00:34:11.000128 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.000013 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t9krq\"" Jul 22 00:34:11.003365 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.003334 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bcc876b87-9kcfx"] Jul 22 00:34:11.004018 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.003987 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 00:34:11.031833 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.031805 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:34:11.168526 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.168496 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-trusted-ca-bundle\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.168652 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.168537 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-serving-cert\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.168652 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.168598 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68b6l\" (UniqueName: \"kubernetes.io/projected/afdd9a9a-c790-479c-a241-adaf589b2ba2-kube-api-access-68b6l\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.168772 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.168666 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-service-ca\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.168772 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.168700 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-oauth-config\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.168772 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.168741 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-config\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.168772 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.168768 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-oauth-serving-cert\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.173267 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.173233 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:34:11.269915 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.269891 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-service-ca\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.270033 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.269924 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-oauth-config\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.270033 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.269949 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-config\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.270033 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.269971 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-oauth-serving-cert\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.270151 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.270097 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-trusted-ca-bundle\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.270202 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.270152 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-serving-cert\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.270272 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.270235 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68b6l\" (UniqueName: \"kubernetes.io/projected/afdd9a9a-c790-479c-a241-adaf589b2ba2-kube-api-access-68b6l\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.270834 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.270811 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-oauth-serving-cert\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.270944 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.270893 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-config\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.271032 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.271002 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-service-ca\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.271292 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.271274 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-trusted-ca-bundle\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.272540 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.272512 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-oauth-config\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.272819 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.272798 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-serving-cert\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.277953 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.277926 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68b6l\" (UniqueName: \"kubernetes.io/projected/afdd9a9a-c790-479c-a241-adaf589b2ba2-kube-api-access-68b6l\") pod \"console-5bcc876b87-9kcfx\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.307526 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.307500 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:11.444291 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.444259 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bcc876b87-9kcfx"] Jul 22 00:34:11.447548 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:34:11.447521 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafdd9a9a_c790_479c_a241_adaf589b2ba2.slice/crio-66e8b850796fadd1372ba003d334aa7179903e82c622079847f359c64a1f739b WatchSource:0}: Error finding container 66e8b850796fadd1372ba003d334aa7179903e82c622079847f359c64a1f739b: Status 404 returned error can't find the container with id 66e8b850796fadd1372ba003d334aa7179903e82c622079847f359c64a1f739b Jul 22 00:34:11.752255 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.752166 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcc876b87-9kcfx" event={"ID":"afdd9a9a-c790-479c-a241-adaf589b2ba2","Type":"ContainerStarted","Data":"bb3b94fef7915d2ade34904d543d7784edcc5e3839f4409d52e85c0198b4109e"} Jul 22 00:34:11.752255 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.752218 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcc876b87-9kcfx" event={"ID":"afdd9a9a-c790-479c-a241-adaf589b2ba2","Type":"ContainerStarted","Data":"66e8b850796fadd1372ba003d334aa7179903e82c622079847f359c64a1f739b"} Jul 22 00:34:11.767625 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:11.767549 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bcc876b87-9kcfx" podStartSLOduration=1.767527337 podStartE2EDuration="1.767527337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:34:11.767269687 +0000 UTC m=+460.093341912" watchObservedRunningTime="2026-07-22 00:34:11.767527337 +0000 UTC m=+460.093599547" Jul 22 00:34:20.284440 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:20.284387 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:34:20.338020 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:20.337987 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:34:21.308648 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:21.308617 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:21.309014 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:21.308667 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:21.313113 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:21.313094 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:21.799362 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:21.799338 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:34:21.937291 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:21.937261 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:34:22.038647 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:22.038612 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:34:22.087830 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:22.087737 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:34:28.441151 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:28.441114 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:34:28.540178 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:28.540143 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:34:28.701203 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:34:28.701118 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:07.381768 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:07.381728 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:14.900915 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:14.900876 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:15.000012 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:14.999975 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:34.739826 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:34.739763 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:41.531142 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:41.531102 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:42.321295 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:42.321260 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:43.439545 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:43.439463 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvqgx"] Jul 22 00:35:43.443052 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:43.443029 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvqgx" Jul 22 00:35:43.445815 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:43.445771 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-k4sqg\"" Jul 22 00:35:43.445815 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:43.445801 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 22 00:35:43.445977 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:43.445816 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 22 00:35:43.456828 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:43.456802 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvqgx"] Jul 22 00:35:43.577992 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:43.577964 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79c75b64-7aff-4ea7-81d9-5f0176597039-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-pvqgx\" (UID: \"79c75b64-7aff-4ea7-81d9-5f0176597039\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvqgx" Jul 22 00:35:43.578091 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:43.578014 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch9ph\" (UniqueName: \"kubernetes.io/projected/79c75b64-7aff-4ea7-81d9-5f0176597039-kube-api-access-ch9ph\") pod \"cert-manager-operator-controller-manager-55c444bf86-pvqgx\" (UID: \"79c75b64-7aff-4ea7-81d9-5f0176597039\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvqgx" Jul 22 00:35:43.679005 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:43.678971 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79c75b64-7aff-4ea7-81d9-5f0176597039-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-pvqgx\" (UID: \"79c75b64-7aff-4ea7-81d9-5f0176597039\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvqgx" Jul 22 00:35:43.679111 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:43.679028 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch9ph\" (UniqueName: \"kubernetes.io/projected/79c75b64-7aff-4ea7-81d9-5f0176597039-kube-api-access-ch9ph\") pod \"cert-manager-operator-controller-manager-55c444bf86-pvqgx\" (UID: \"79c75b64-7aff-4ea7-81d9-5f0176597039\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvqgx" Jul 22 00:35:43.679384 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:43.679365 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79c75b64-7aff-4ea7-81d9-5f0176597039-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-pvqgx\" (UID: \"79c75b64-7aff-4ea7-81d9-5f0176597039\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvqgx" Jul 22 00:35:43.687586 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:43.687555 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch9ph\" (UniqueName: \"kubernetes.io/projected/79c75b64-7aff-4ea7-81d9-5f0176597039-kube-api-access-ch9ph\") pod \"cert-manager-operator-controller-manager-55c444bf86-pvqgx\" (UID: \"79c75b64-7aff-4ea7-81d9-5f0176597039\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvqgx" Jul 22 00:35:43.753068 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:43.753014 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvqgx" Jul 22 00:35:43.884437 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:43.884411 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvqgx"] Jul 22 00:35:43.890344 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:35:43.890306 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c75b64_7aff_4ea7_81d9_5f0176597039.slice/crio-b72f1197208496610f263fd4049299a33f3f4d98f169b1b886011f8193c1647a WatchSource:0}: Error finding container b72f1197208496610f263fd4049299a33f3f4d98f169b1b886011f8193c1647a: Status 404 returned error can't find the container with id b72f1197208496610f263fd4049299a33f3f4d98f169b1b886011f8193c1647a Jul 22 00:35:44.120012 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:44.119979 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvqgx" event={"ID":"79c75b64-7aff-4ea7-81d9-5f0176597039","Type":"ContainerStarted","Data":"b72f1197208496610f263fd4049299a33f3f4d98f169b1b886011f8193c1647a"} Jul 22 00:35:45.435827 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:45.435768 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:45.745758 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:45.745663 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:47.984562 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:47.984524 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lm22q"] Jul 22 00:35:47.987902 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:47.987886 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lm22q" Jul 22 00:35:47.990541 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:47.990518 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 00:35:47.990655 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:47.990563 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-z7jgf\"" Jul 22 00:35:47.990655 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:47.990638 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 00:35:47.998327 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:47.998303 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lm22q"] Jul 22 00:35:48.045758 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:48.045727 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:48.117825 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:48.117795 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc42bc95-ce24-4001-b22a-c47c3cd26c09-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lm22q\" (UID: \"fc42bc95-ce24-4001-b22a-c47c3cd26c09\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lm22q" Jul 22 00:35:48.117974 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:48.117830 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv7sn\" (UniqueName: \"kubernetes.io/projected/fc42bc95-ce24-4001-b22a-c47c3cd26c09-kube-api-access-nv7sn\") pod \"cert-manager-webhook-6486ffd997-lm22q\" (UID: \"fc42bc95-ce24-4001-b22a-c47c3cd26c09\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lm22q" Jul 22 00:35:48.139746 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:48.139716 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvqgx" event={"ID":"79c75b64-7aff-4ea7-81d9-5f0176597039","Type":"ContainerStarted","Data":"0e63efef55a1c6edf4435e12ab7b184121521693e29e7c3891606daa99df2813"} Jul 22 00:35:48.158703 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:48.158647 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvqgx" podStartSLOduration=1.941343077 podStartE2EDuration="5.158627644s" totalImagesPullingTime="3.217284567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:35:48.158003597 +0000 UTC m=+556.484075798" watchObservedRunningTime="2026-07-22 00:35:48.158627644 +0000 UTC m=+556.484699847" Jul 22 00:35:48.180759 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:48.180695 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:48.218899 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:48.218871 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc42bc95-ce24-4001-b22a-c47c3cd26c09-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lm22q\" (UID: \"fc42bc95-ce24-4001-b22a-c47c3cd26c09\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lm22q" Jul 22 00:35:48.218997 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:48.218907 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv7sn\" (UniqueName: \"kubernetes.io/projected/fc42bc95-ce24-4001-b22a-c47c3cd26c09-kube-api-access-nv7sn\") pod \"cert-manager-webhook-6486ffd997-lm22q\" (UID: \"fc42bc95-ce24-4001-b22a-c47c3cd26c09\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lm22q" Jul 22 00:35:48.227251 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:48.227230 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc42bc95-ce24-4001-b22a-c47c3cd26c09-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lm22q\" (UID: \"fc42bc95-ce24-4001-b22a-c47c3cd26c09\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lm22q" Jul 22 00:35:48.227428 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:48.227411 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv7sn\" (UniqueName: \"kubernetes.io/projected/fc42bc95-ce24-4001-b22a-c47c3cd26c09-kube-api-access-nv7sn\") pod \"cert-manager-webhook-6486ffd997-lm22q\" (UID: \"fc42bc95-ce24-4001-b22a-c47c3cd26c09\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lm22q" Jul 22 00:35:48.309126 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:48.309104 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lm22q" Jul 22 00:35:48.446976 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:48.446953 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lm22q"] Jul 22 00:35:48.449531 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:35:48.449501 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc42bc95_ce24_4001_b22a_c47c3cd26c09.slice/crio-c97fda41a69c8c83bc396a752a6b493bfad477197d0dd10777ba45ce794c63a4 WatchSource:0}: Error finding container c97fda41a69c8c83bc396a752a6b493bfad477197d0dd10777ba45ce794c63a4: Status 404 returned error can't find the container with id c97fda41a69c8c83bc396a752a6b493bfad477197d0dd10777ba45ce794c63a4 Jul 22 00:35:48.491889 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:48.491864 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:48.648523 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:48.648457 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:49.144708 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:49.144669 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lm22q" event={"ID":"fc42bc95-ce24-4001-b22a-c47c3cd26c09","Type":"ContainerStarted","Data":"c97fda41a69c8c83bc396a752a6b493bfad477197d0dd10777ba45ce794c63a4"} Jul 22 00:35:49.810876 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:49.810832 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:50.206867 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:50.206759 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:50.376466 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:50.376439 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bvqd5"] Jul 22 00:35:50.379634 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:50.379618 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bvqd5" Jul 22 00:35:50.382349 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:50.382330 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-c8jls\"" Jul 22 00:35:50.386718 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:50.386695 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bvqd5"] Jul 22 00:35:50.540460 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:50.540430 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsh7s\" (UniqueName: \"kubernetes.io/projected/576dfd12-4b1f-4f3a-8874-fa0be12bec89-kube-api-access-hsh7s\") pod \"cert-manager-cainjector-559cdd8c87-bvqd5\" (UID: \"576dfd12-4b1f-4f3a-8874-fa0be12bec89\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bvqd5" Jul 22 00:35:50.540596 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:50.540502 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/576dfd12-4b1f-4f3a-8874-fa0be12bec89-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bvqd5\" (UID: \"576dfd12-4b1f-4f3a-8874-fa0be12bec89\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bvqd5" Jul 22 00:35:50.641476 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:50.641446 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsh7s\" (UniqueName: \"kubernetes.io/projected/576dfd12-4b1f-4f3a-8874-fa0be12bec89-kube-api-access-hsh7s\") pod \"cert-manager-cainjector-559cdd8c87-bvqd5\" (UID: \"576dfd12-4b1f-4f3a-8874-fa0be12bec89\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bvqd5" Jul 22 00:35:50.641605 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:50.641525 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/576dfd12-4b1f-4f3a-8874-fa0be12bec89-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bvqd5\" (UID: \"576dfd12-4b1f-4f3a-8874-fa0be12bec89\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bvqd5" Jul 22 00:35:50.650871 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:50.650841 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/576dfd12-4b1f-4f3a-8874-fa0be12bec89-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bvqd5\" (UID: \"576dfd12-4b1f-4f3a-8874-fa0be12bec89\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bvqd5" Jul 22 00:35:50.651247 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:50.651226 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsh7s\" (UniqueName: \"kubernetes.io/projected/576dfd12-4b1f-4f3a-8874-fa0be12bec89-kube-api-access-hsh7s\") pod \"cert-manager-cainjector-559cdd8c87-bvqd5\" (UID: \"576dfd12-4b1f-4f3a-8874-fa0be12bec89\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bvqd5" Jul 22 00:35:50.690339 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:50.690318 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bvqd5" Jul 22 00:35:50.831438 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:50.831403 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bvqd5"] Jul 22 00:35:50.833732 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:35:50.833704 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod576dfd12_4b1f_4f3a_8874_fa0be12bec89.slice/crio-ad1c7d0947d0db522a0785108291be4c179717c432f88027c19e5c7ec5a86a19 WatchSource:0}: Error finding container ad1c7d0947d0db522a0785108291be4c179717c432f88027c19e5c7ec5a86a19: Status 404 returned error can't find the container with id ad1c7d0947d0db522a0785108291be4c179717c432f88027c19e5c7ec5a86a19 Jul 22 00:35:50.932932 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:50.932898 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:51.032631 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:51.032594 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:35:51.160353 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:51.160260 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bvqd5" event={"ID":"576dfd12-4b1f-4f3a-8874-fa0be12bec89","Type":"ContainerStarted","Data":"ad1c7d0947d0db522a0785108291be4c179717c432f88027c19e5c7ec5a86a19"} Jul 22 00:35:52.166040 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:52.165996 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lm22q" event={"ID":"fc42bc95-ce24-4001-b22a-c47c3cd26c09","Type":"ContainerStarted","Data":"f5874a42f11f23596dd0e912c1d3d433105ea0ba066df4b01d3104f0f1fa20f9"} Jul 22 00:35:52.166531 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:52.166215 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lm22q" Jul 22 00:35:52.168942 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:52.168865 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bvqd5" event={"ID":"576dfd12-4b1f-4f3a-8874-fa0be12bec89","Type":"ContainerStarted","Data":"b5c69e5d0330dfd60a5c755c6f8a5c67ad0bcaaa9eaaa6d7112e061f4633d96f"} Jul 22 00:35:52.202161 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:52.202095 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-lm22q" podStartSLOduration=1.726722963 podStartE2EDuration="5.202075417s" totalImagesPullingTime="3.475352454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:35:52.200462686 +0000 UTC m=+560.526534888" watchObservedRunningTime="2026-07-22 00:35:52.202075417 +0000 UTC m=+560.528147622" Jul 22 00:35:52.224758 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:52.224714 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bvqd5" podStartSLOduration=1.133123774 podStartE2EDuration="2.224700976s" totalImagesPullingTime="1.091577202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:35:52.224248395 +0000 UTC m=+560.550320598" watchObservedRunningTime="2026-07-22 00:35:52.224700976 +0000 UTC m=+560.550773180" Jul 22 00:35:58.176949 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:58.176905 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lm22q" Jul 22 00:35:59.027680 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:35:59.027636 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:10.550523 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:10.550477 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:10.646234 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:10.646185 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:15.684637 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:15.684574 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:15.784935 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:15.784906 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:18.047285 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:18.047245 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:18.892988 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:18.892947 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:18.950026 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:18.949976 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:24.265902 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.265862 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:24.331599 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.331565 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8"] Jul 22 00:36:24.338273 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.338251 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:24.342093 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.342069 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 22 00:36:24.342212 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.342101 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 22 00:36:24.342363 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.342343 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 22 00:36:24.342671 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.342492 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 00:36:24.342947 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.342927 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-j22h7\"" Jul 22 00:36:24.343062 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.343041 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 00:36:24.344456 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.344430 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8"] Jul 22 00:36:24.351065 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.351037 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:24.471787 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.471755 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d17de76-1014-4177-b623-b04844ccb839-metrics-cert\") pod \"lws-controller-manager-7bd6f87dc6-pnvb8\" (UID: \"1d17de76-1014-4177-b623-b04844ccb839\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:24.471929 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.471856 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d17de76-1014-4177-b623-b04844ccb839-manager-config\") pod \"lws-controller-manager-7bd6f87dc6-pnvb8\" (UID: \"1d17de76-1014-4177-b623-b04844ccb839\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:24.471929 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.471876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d17de76-1014-4177-b623-b04844ccb839-cert\") pod \"lws-controller-manager-7bd6f87dc6-pnvb8\" (UID: \"1d17de76-1014-4177-b623-b04844ccb839\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:24.471929 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.471898 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnqcx\" (UniqueName: \"kubernetes.io/projected/1d17de76-1014-4177-b623-b04844ccb839-kube-api-access-tnqcx\") pod \"lws-controller-manager-7bd6f87dc6-pnvb8\" (UID: \"1d17de76-1014-4177-b623-b04844ccb839\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:24.572818 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.572733 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d17de76-1014-4177-b623-b04844ccb839-metrics-cert\") pod \"lws-controller-manager-7bd6f87dc6-pnvb8\" (UID: \"1d17de76-1014-4177-b623-b04844ccb839\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:24.572935 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.572826 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d17de76-1014-4177-b623-b04844ccb839-manager-config\") pod \"lws-controller-manager-7bd6f87dc6-pnvb8\" (UID: \"1d17de76-1014-4177-b623-b04844ccb839\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:24.572935 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.572855 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d17de76-1014-4177-b623-b04844ccb839-cert\") pod \"lws-controller-manager-7bd6f87dc6-pnvb8\" (UID: \"1d17de76-1014-4177-b623-b04844ccb839\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:24.572935 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.572894 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnqcx\" (UniqueName: \"kubernetes.io/projected/1d17de76-1014-4177-b623-b04844ccb839-kube-api-access-tnqcx\") pod \"lws-controller-manager-7bd6f87dc6-pnvb8\" (UID: \"1d17de76-1014-4177-b623-b04844ccb839\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:24.573610 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.573580 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d17de76-1014-4177-b623-b04844ccb839-manager-config\") pod \"lws-controller-manager-7bd6f87dc6-pnvb8\" (UID: \"1d17de76-1014-4177-b623-b04844ccb839\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:24.575347 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.575314 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d17de76-1014-4177-b623-b04844ccb839-metrics-cert\") pod \"lws-controller-manager-7bd6f87dc6-pnvb8\" (UID: \"1d17de76-1014-4177-b623-b04844ccb839\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:24.575347 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.575336 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d17de76-1014-4177-b623-b04844ccb839-cert\") pod \"lws-controller-manager-7bd6f87dc6-pnvb8\" (UID: \"1d17de76-1014-4177-b623-b04844ccb839\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:24.580871 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.580851 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnqcx\" (UniqueName: \"kubernetes.io/projected/1d17de76-1014-4177-b623-b04844ccb839-kube-api-access-tnqcx\") pod \"lws-controller-manager-7bd6f87dc6-pnvb8\" (UID: \"1d17de76-1014-4177-b623-b04844ccb839\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:24.648919 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.648891 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:24.774515 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:24.774491 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8"] Jul 22 00:36:24.776957 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:36:24.776929 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d17de76_1014_4177_b623_b04844ccb839.slice/crio-bbc1ad5181faa178ba75ba6f586289ed9c15b2d4166d70ecc45972ba8f68fc28 WatchSource:0}: Error finding container bbc1ad5181faa178ba75ba6f586289ed9c15b2d4166d70ecc45972ba8f68fc28: Status 404 returned error can't find the container with id bbc1ad5181faa178ba75ba6f586289ed9c15b2d4166d70ecc45972ba8f68fc28 Jul 22 00:36:25.316485 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:25.316452 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" event={"ID":"1d17de76-1014-4177-b623-b04844ccb839","Type":"ContainerStarted","Data":"bbc1ad5181faa178ba75ba6f586289ed9c15b2d4166d70ecc45972ba8f68fc28"} Jul 22 00:36:27.332408 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:27.332373 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" event={"ID":"1d17de76-1014-4177-b623-b04844ccb839","Type":"ContainerStarted","Data":"bef682ae576858e86b840545b21026659eaa69e5d5564db35694200e3384ba5d"} Jul 22 00:36:27.332734 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:27.332450 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:27.349141 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:27.349103 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" podStartSLOduration=0.884808238 podStartE2EDuration="3.349088006s" totalImagesPullingTime="2.464279768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:36:27.347493846 +0000 UTC m=+595.673566048" watchObservedRunningTime="2026-07-22 00:36:27.349088006 +0000 UTC m=+595.675160279" Jul 22 00:36:34.342268 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:34.342232 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:34.436272 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:34.436226 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:34.953164 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:34.953127 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:35.187141 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.187111 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:35.294463 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.294427 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:35.321817 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.321769 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh"] Jul 22 00:36:35.325583 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.325561 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.328707 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.328683 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 00:36:35.329058 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.329028 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 00:36:35.329393 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.329374 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 00:36:35.329517 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.329495 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 00:36:35.329596 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.329562 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8wt49\"" Jul 22 00:36:35.345738 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.345710 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh"] Jul 22 00:36:35.352381 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.352351 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg7rf\" (UniqueName: \"kubernetes.io/projected/c5e7758c-a583-43fc-b0d3-06855c41cbbd-kube-api-access-lg7rf\") pod \"opendatahub-operator-controller-manager-6bb84b476f-vg4gh\" (UID: \"c5e7758c-a583-43fc-b0d3-06855c41cbbd\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.352496 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.352430 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5e7758c-a583-43fc-b0d3-06855c41cbbd-tmp\") pod \"opendatahub-operator-controller-manager-6bb84b476f-vg4gh\" (UID: \"c5e7758c-a583-43fc-b0d3-06855c41cbbd\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.352496 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.352458 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5e7758c-a583-43fc-b0d3-06855c41cbbd-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-vg4gh\" (UID: \"c5e7758c-a583-43fc-b0d3-06855c41cbbd\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.352496 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.352486 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c5e7758c-a583-43fc-b0d3-06855c41cbbd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-vg4gh\" (UID: \"c5e7758c-a583-43fc-b0d3-06855c41cbbd\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.352616 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.352576 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c5e7758c-a583-43fc-b0d3-06855c41cbbd-manifests\") pod \"opendatahub-operator-controller-manager-6bb84b476f-vg4gh\" (UID: \"c5e7758c-a583-43fc-b0d3-06855c41cbbd\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.452905 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.452870 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg7rf\" (UniqueName: \"kubernetes.io/projected/c5e7758c-a583-43fc-b0d3-06855c41cbbd-kube-api-access-lg7rf\") pod \"opendatahub-operator-controller-manager-6bb84b476f-vg4gh\" (UID: \"c5e7758c-a583-43fc-b0d3-06855c41cbbd\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.453070 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.452945 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5e7758c-a583-43fc-b0d3-06855c41cbbd-tmp\") pod \"opendatahub-operator-controller-manager-6bb84b476f-vg4gh\" (UID: \"c5e7758c-a583-43fc-b0d3-06855c41cbbd\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.453070 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.452972 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5e7758c-a583-43fc-b0d3-06855c41cbbd-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-vg4gh\" (UID: \"c5e7758c-a583-43fc-b0d3-06855c41cbbd\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.453070 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.453002 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c5e7758c-a583-43fc-b0d3-06855c41cbbd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-vg4gh\" (UID: \"c5e7758c-a583-43fc-b0d3-06855c41cbbd\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.453070 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.453060 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c5e7758c-a583-43fc-b0d3-06855c41cbbd-manifests\") pod \"opendatahub-operator-controller-manager-6bb84b476f-vg4gh\" (UID: \"c5e7758c-a583-43fc-b0d3-06855c41cbbd\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.453461 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.453435 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c5e7758c-a583-43fc-b0d3-06855c41cbbd-manifests\") pod \"opendatahub-operator-controller-manager-6bb84b476f-vg4gh\" (UID: \"c5e7758c-a583-43fc-b0d3-06855c41cbbd\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.455362 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.455335 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5e7758c-a583-43fc-b0d3-06855c41cbbd-tmp\") pod \"opendatahub-operator-controller-manager-6bb84b476f-vg4gh\" (UID: \"c5e7758c-a583-43fc-b0d3-06855c41cbbd\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.455509 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.455490 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5e7758c-a583-43fc-b0d3-06855c41cbbd-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-vg4gh\" (UID: \"c5e7758c-a583-43fc-b0d3-06855c41cbbd\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.455673 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.455656 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c5e7758c-a583-43fc-b0d3-06855c41cbbd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-vg4gh\" (UID: \"c5e7758c-a583-43fc-b0d3-06855c41cbbd\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.461355 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.461330 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg7rf\" (UniqueName: \"kubernetes.io/projected/c5e7758c-a583-43fc-b0d3-06855c41cbbd-kube-api-access-lg7rf\") pod \"opendatahub-operator-controller-manager-6bb84b476f-vg4gh\" (UID: \"c5e7758c-a583-43fc-b0d3-06855c41cbbd\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.492926 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.492891 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:35.589863 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.589791 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:35.637328 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.637303 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:35.691572 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.691543 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:35.776161 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.776131 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh"] Jul 22 00:36:35.778546 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:36:35.778523 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e7758c_a583_43fc_b0d3_06855c41cbbd.slice/crio-f26b403272e8e4d5264c114db9c23cd55d22bc3a04e81b6fc2704ef09b67650b WatchSource:0}: Error finding container f26b403272e8e4d5264c114db9c23cd55d22bc3a04e81b6fc2704ef09b67650b: Status 404 returned error can't find the container with id f26b403272e8e4d5264c114db9c23cd55d22bc3a04e81b6fc2704ef09b67650b Jul 22 00:36:35.785401 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.785364 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:35.871445 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.871383 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:35.972219 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:35.972189 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:36.373169 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:36.373138 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" event={"ID":"c5e7758c-a583-43fc-b0d3-06855c41cbbd","Type":"ContainerStarted","Data":"f26b403272e8e4d5264c114db9c23cd55d22bc3a04e81b6fc2704ef09b67650b"} Jul 22 00:36:38.339603 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:38.339575 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-pnvb8" Jul 22 00:36:39.390349 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:39.390323 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" event={"ID":"c5e7758c-a583-43fc-b0d3-06855c41cbbd","Type":"ContainerStarted","Data":"5e865a240ea3ea6eb17ca236081117a0d4669a5e64356795e5b46ccf02d6be33"} Jul 22 00:36:40.054377 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:40.054352 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:36:40.055401 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:40.055379 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:36:40.395679 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:40.395651 2567 generic.go:356] "Generic (PLEG): container finished" podID="c5e7758c-a583-43fc-b0d3-06855c41cbbd" containerID="5e865a240ea3ea6eb17ca236081117a0d4669a5e64356795e5b46ccf02d6be33" exitCode=0 Jul 22 00:36:40.396010 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:40.395750 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" event={"ID":"c5e7758c-a583-43fc-b0d3-06855c41cbbd","Type":"ContainerDied","Data":"5e865a240ea3ea6eb17ca236081117a0d4669a5e64356795e5b46ccf02d6be33"} Jul 22 00:36:40.396161 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:40.396146 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 00:36:41.401600 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:41.401503 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" event={"ID":"c5e7758c-a583-43fc-b0d3-06855c41cbbd","Type":"ContainerStarted","Data":"e70f761e952abeb084a9321aec76a5486fd02a3334eaeb253f66528f06f575bf"} Jul 22 00:36:41.401600 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:41.401572 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:41.422422 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:41.422382 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" podStartSLOduration=1.307102918 podStartE2EDuration="6.422368669s" totalImagesPullingTime="4.093766335s" totalInitContainerRuntime="1.021499416s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:36:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:36:41.421733247 +0000 UTC m=+609.747805475" watchObservedRunningTime="2026-07-22 00:36:41.422368669 +0000 UTC m=+609.748440943" Jul 22 00:36:44.910510 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:44.910475 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:46.922370 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:46.922329 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:47.023880 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:47.023841 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:47.357241 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:47.357210 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:47.456675 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:47.456646 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:51.338183 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:51.338148 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:51.389191 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:51.389158 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:51.801712 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:51.801681 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:51.861832 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:51.861801 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:52.407673 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:52.407649 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-vg4gh" Jul 22 00:36:54.783257 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:54.783163 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:55.338887 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:55.338839 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:55.536198 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:55.536159 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:55.613941 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:55.613854 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:55.714979 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:55.714939 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:55.851796 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:55.851747 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:56.156614 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:56.156582 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:36:56.612323 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:36:56.612288 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:00.628347 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.628312 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:00.704681 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.704647 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq"] Jul 22 00:37:00.708310 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.708288 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq" Jul 22 00:37:00.710758 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.710729 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 00:37:00.710941 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.710834 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-95fht\"" Jul 22 00:37:00.711102 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.711083 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 00:37:00.718132 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.718111 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq"] Jul 22 00:37:00.752959 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.752933 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skhp4\" (UniqueName: \"kubernetes.io/projected/004b59a0-37e8-4897-9f04-58797d56b611-kube-api-access-skhp4\") pod \"kube-auth-proxy-6467ff7b5c-vxdwq\" (UID: \"004b59a0-37e8-4897-9f04-58797d56b611\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq" Jul 22 00:37:00.753072 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.752965 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/004b59a0-37e8-4897-9f04-58797d56b611-tls-certs\") pod \"kube-auth-proxy-6467ff7b5c-vxdwq\" (UID: \"004b59a0-37e8-4897-9f04-58797d56b611\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq" Jul 22 00:37:00.753072 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.753002 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/004b59a0-37e8-4897-9f04-58797d56b611-tmp\") pod \"kube-auth-proxy-6467ff7b5c-vxdwq\" (UID: \"004b59a0-37e8-4897-9f04-58797d56b611\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq" Jul 22 00:37:00.854486 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.854446 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/004b59a0-37e8-4897-9f04-58797d56b611-tmp\") pod \"kube-auth-proxy-6467ff7b5c-vxdwq\" (UID: \"004b59a0-37e8-4897-9f04-58797d56b611\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq" Jul 22 00:37:00.854649 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.854584 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skhp4\" (UniqueName: \"kubernetes.io/projected/004b59a0-37e8-4897-9f04-58797d56b611-kube-api-access-skhp4\") pod \"kube-auth-proxy-6467ff7b5c-vxdwq\" (UID: \"004b59a0-37e8-4897-9f04-58797d56b611\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq" Jul 22 00:37:00.854649 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.854612 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/004b59a0-37e8-4897-9f04-58797d56b611-tls-certs\") pod \"kube-auth-proxy-6467ff7b5c-vxdwq\" (UID: \"004b59a0-37e8-4897-9f04-58797d56b611\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq" Jul 22 00:37:00.857100 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.857070 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/004b59a0-37e8-4897-9f04-58797d56b611-tmp\") pod \"kube-auth-proxy-6467ff7b5c-vxdwq\" (UID: \"004b59a0-37e8-4897-9f04-58797d56b611\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq" Jul 22 00:37:00.857240 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.857114 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:00.857374 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.857356 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/004b59a0-37e8-4897-9f04-58797d56b611-tls-certs\") pod \"kube-auth-proxy-6467ff7b5c-vxdwq\" (UID: \"004b59a0-37e8-4897-9f04-58797d56b611\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq" Jul 22 00:37:00.861789 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:00.861755 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skhp4\" (UniqueName: \"kubernetes.io/projected/004b59a0-37e8-4897-9f04-58797d56b611-kube-api-access-skhp4\") pod \"kube-auth-proxy-6467ff7b5c-vxdwq\" (UID: \"004b59a0-37e8-4897-9f04-58797d56b611\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq" Jul 22 00:37:01.019340 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:01.019314 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq" Jul 22 00:37:01.154306 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:01.154279 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq"] Jul 22 00:37:01.157101 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:37:01.157068 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004b59a0_37e8_4897_9f04_58797d56b611.slice/crio-e6a936976a923dbb3fd1b380eb0ae574fbacd542571036bc040644fa6f6be0b9 WatchSource:0}: Error finding container e6a936976a923dbb3fd1b380eb0ae574fbacd542571036bc040644fa6f6be0b9: Status 404 returned error can't find the container with id e6a936976a923dbb3fd1b380eb0ae574fbacd542571036bc040644fa6f6be0b9 Jul 22 00:37:01.495615 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:01.495535 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq" event={"ID":"004b59a0-37e8-4897-9f04-58797d56b611","Type":"ContainerStarted","Data":"e6a936976a923dbb3fd1b380eb0ae574fbacd542571036bc040644fa6f6be0b9"} Jul 22 00:37:01.987634 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:01.987599 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:02.108339 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:02.108305 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:03.206124 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:03.206085 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:04.515284 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:04.515244 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq" event={"ID":"004b59a0-37e8-4897-9f04-58797d56b611","Type":"ContainerStarted","Data":"a1c47d19a31af83dde5a286e776f516e437a57111e14d1243ce14eacd1b4e169"} Jul 22 00:37:04.539080 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:04.539032 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-vxdwq" podStartSLOduration=1.5469223410000001 podStartE2EDuration="4.539017175s" totalImagesPullingTime="2.992094834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:37:04.536141961 +0000 UTC m=+632.862214177" watchObservedRunningTime="2026-07-22 00:37:04.539017175 +0000 UTC m=+632.865089377" Jul 22 00:37:04.822504 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:04.822411 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:04.912159 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:04.912123 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:05.202195 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:05.202116 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:05.382549 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:05.382520 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:06.067489 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:06.067455 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:06.177135 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:06.177089 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:06.282298 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:06.282267 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:07.088121 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:07.088081 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:07.161698 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:07.161663 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:08.048180 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:08.048139 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:08.202363 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:08.202323 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:08.296878 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:08.296845 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:10.388768 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:10.388727 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:10.444540 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:10.444505 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:15.804957 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:15.804920 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:56.239436 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:56.239402 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:56.303290 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:56.303263 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:56.406615 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:56.406578 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:58.131729 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:58.131694 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:37:58.231220 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:37:58.231182 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:09.776038 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:09.775984 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:09.923989 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:09.923954 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:10.520161 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:10.520123 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:10.569463 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:10.569428 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:20.709060 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:20.709019 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bcc876b87-9kcfx"] Jul 22 00:38:20.807232 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:20.807178 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:20.857302 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:20.857261 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:20.911210 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:20.911175 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:26.105344 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:26.105253 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:29.720160 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:29.720118 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:32.750726 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:32.750690 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:34.269478 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:34.269448 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:34.524269 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:34.524186 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:35.293096 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:35.293059 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:37.563382 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:37.563345 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:38.512409 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:38.512349 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:40.383538 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:40.383493 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:41.935771 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:41.935732 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:42.036707 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:42.036669 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:43.143485 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.143446 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt"] Jul 22 00:38:43.146978 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.146952 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt" Jul 22 00:38:43.149624 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.149594 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 22 00:38:43.149794 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.149608 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 22 00:38:43.149904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.149678 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mphx6\"" Jul 22 00:38:43.149904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.149689 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 00:38:43.150017 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.149725 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 00:38:43.155602 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.155577 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt"] Jul 22 00:38:43.161009 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.160982 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:43.262963 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.262936 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf7vt\" (UniqueName: \"kubernetes.io/projected/6339b835-255f-4abf-90c3-8a26cee5522f-kube-api-access-cf7vt\") pod \"kuadrant-console-plugin-66b5774985-8jlvt\" (UID: \"6339b835-255f-4abf-90c3-8a26cee5522f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt" Jul 22 00:38:43.263081 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.263028 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6339b835-255f-4abf-90c3-8a26cee5522f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8jlvt\" (UID: \"6339b835-255f-4abf-90c3-8a26cee5522f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt" Jul 22 00:38:43.263081 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.263056 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6339b835-255f-4abf-90c3-8a26cee5522f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8jlvt\" (UID: \"6339b835-255f-4abf-90c3-8a26cee5522f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt" Jul 22 00:38:43.346596 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.346570 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:43.363608 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.363580 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7vt\" (UniqueName: \"kubernetes.io/projected/6339b835-255f-4abf-90c3-8a26cee5522f-kube-api-access-cf7vt\") pod \"kuadrant-console-plugin-66b5774985-8jlvt\" (UID: \"6339b835-255f-4abf-90c3-8a26cee5522f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt" Jul 22 00:38:43.363707 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.363623 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6339b835-255f-4abf-90c3-8a26cee5522f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8jlvt\" (UID: \"6339b835-255f-4abf-90c3-8a26cee5522f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt" Jul 22 00:38:43.363707 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.363640 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6339b835-255f-4abf-90c3-8a26cee5522f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8jlvt\" (UID: \"6339b835-255f-4abf-90c3-8a26cee5522f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt" Jul 22 00:38:43.364233 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.364212 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6339b835-255f-4abf-90c3-8a26cee5522f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8jlvt\" (UID: \"6339b835-255f-4abf-90c3-8a26cee5522f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt" Jul 22 00:38:43.366025 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.366004 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6339b835-255f-4abf-90c3-8a26cee5522f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8jlvt\" (UID: \"6339b835-255f-4abf-90c3-8a26cee5522f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt" Jul 22 00:38:43.370730 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.370713 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf7vt\" (UniqueName: \"kubernetes.io/projected/6339b835-255f-4abf-90c3-8a26cee5522f-kube-api-access-cf7vt\") pod \"kuadrant-console-plugin-66b5774985-8jlvt\" (UID: \"6339b835-255f-4abf-90c3-8a26cee5522f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt" Jul 22 00:38:43.457591 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.457544 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt" Jul 22 00:38:43.587884 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.587826 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt"] Jul 22 00:38:43.590155 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:38:43.590126 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6339b835_255f_4abf_90c3_8a26cee5522f.slice/crio-7d1cb506329d9c0b770886e85ee832a01717ecf777f027b390a87db052cd9993 WatchSource:0}: Error finding container 7d1cb506329d9c0b770886e85ee832a01717ecf777f027b390a87db052cd9993: Status 404 returned error can't find the container with id 7d1cb506329d9c0b770886e85ee832a01717ecf777f027b390a87db052cd9993 Jul 22 00:38:43.993616 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:43.993585 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt" event={"ID":"6339b835-255f-4abf-90c3-8a26cee5522f","Type":"ContainerStarted","Data":"7d1cb506329d9c0b770886e85ee832a01717ecf777f027b390a87db052cd9993"} Jul 22 00:38:45.510646 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:45.510592 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:45.611011 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:45.610948 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:45.733485 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:45.733438 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bcc876b87-9kcfx" podUID="afdd9a9a-c790-479c-a241-adaf589b2ba2" containerName="console" containerID="cri-o://bb3b94fef7915d2ade34904d543d7784edcc5e3839f4409d52e85c0198b4109e" gracePeriod=15 Jul 22 00:38:46.008061 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.005894 2567 generic.go:356] "Generic (PLEG): container finished" podID="afdd9a9a-c790-479c-a241-adaf589b2ba2" containerID="bb3b94fef7915d2ade34904d543d7784edcc5e3839f4409d52e85c0198b4109e" exitCode=0 Jul 22 00:38:46.008061 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.005975 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcc876b87-9kcfx" event={"ID":"afdd9a9a-c790-479c-a241-adaf589b2ba2","Type":"ContainerDied","Data":"bb3b94fef7915d2ade34904d543d7784edcc5e3839f4409d52e85c0198b4109e"} Jul 22 00:38:46.008061 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.006020 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcc876b87-9kcfx" event={"ID":"afdd9a9a-c790-479c-a241-adaf589b2ba2","Type":"ContainerDied","Data":"66e8b850796fadd1372ba003d334aa7179903e82c622079847f359c64a1f739b"} Jul 22 00:38:46.008061 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.006041 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66e8b850796fadd1372ba003d334aa7179903e82c622079847f359c64a1f739b" Jul 22 00:38:46.029617 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.029523 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:38:46.187742 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.187615 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-trusted-ca-bundle\") pod \"afdd9a9a-c790-479c-a241-adaf589b2ba2\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " Jul 22 00:38:46.187742 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.187693 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-service-ca\") pod \"afdd9a9a-c790-479c-a241-adaf589b2ba2\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " Jul 22 00:38:46.187742 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.187731 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68b6l\" (UniqueName: \"kubernetes.io/projected/afdd9a9a-c790-479c-a241-adaf589b2ba2-kube-api-access-68b6l\") pod \"afdd9a9a-c790-479c-a241-adaf589b2ba2\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " Jul 22 00:38:46.188060 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.187756 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-oauth-serving-cert\") pod \"afdd9a9a-c790-479c-a241-adaf589b2ba2\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " Jul 22 00:38:46.188060 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.187865 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-oauth-config\") pod \"afdd9a9a-c790-479c-a241-adaf589b2ba2\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " Jul 22 00:38:46.188060 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.187903 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-serving-cert\") pod \"afdd9a9a-c790-479c-a241-adaf589b2ba2\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " Jul 22 00:38:46.188060 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.187928 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-config\") pod \"afdd9a9a-c790-479c-a241-adaf589b2ba2\" (UID: \"afdd9a9a-c790-479c-a241-adaf589b2ba2\") " Jul 22 00:38:46.189436 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.188487 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-config" (OuterVolumeSpecName: "console-config") pod "afdd9a9a-c790-479c-a241-adaf589b2ba2" (UID: "afdd9a9a-c790-479c-a241-adaf589b2ba2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:38:46.189436 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.188962 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "afdd9a9a-c790-479c-a241-adaf589b2ba2" (UID: "afdd9a9a-c790-479c-a241-adaf589b2ba2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:38:46.189436 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.189318 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-service-ca" (OuterVolumeSpecName: "service-ca") pod "afdd9a9a-c790-479c-a241-adaf589b2ba2" (UID: "afdd9a9a-c790-479c-a241-adaf589b2ba2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:38:46.189924 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.189878 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "afdd9a9a-c790-479c-a241-adaf589b2ba2" (UID: "afdd9a9a-c790-479c-a241-adaf589b2ba2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:38:46.192824 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.192773 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "afdd9a9a-c790-479c-a241-adaf589b2ba2" (UID: "afdd9a9a-c790-479c-a241-adaf589b2ba2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:38:46.193289 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.193267 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "afdd9a9a-c790-479c-a241-adaf589b2ba2" (UID: "afdd9a9a-c790-479c-a241-adaf589b2ba2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:38:46.194003 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.193967 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afdd9a9a-c790-479c-a241-adaf589b2ba2-kube-api-access-68b6l" (OuterVolumeSpecName: "kube-api-access-68b6l") pod "afdd9a9a-c790-479c-a241-adaf589b2ba2" (UID: "afdd9a9a-c790-479c-a241-adaf589b2ba2"). InnerVolumeSpecName "kube-api-access-68b6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:38:46.289635 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.289606 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-oauth-config\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:38:46.289747 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.289638 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-serving-cert\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:38:46.289747 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.289652 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-console-config\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:38:46.289747 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.289666 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-trusted-ca-bundle\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:38:46.289747 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.289680 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-service-ca\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:38:46.289747 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.289696 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68b6l\" (UniqueName: \"kubernetes.io/projected/afdd9a9a-c790-479c-a241-adaf589b2ba2-kube-api-access-68b6l\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:38:46.289747 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:46.289707 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afdd9a9a-c790-479c-a241-adaf589b2ba2-oauth-serving-cert\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:38:47.010892 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:47.010859 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcc876b87-9kcfx" Jul 22 00:38:47.037095 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:47.037068 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bcc876b87-9kcfx"] Jul 22 00:38:47.043660 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:47.043636 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bcc876b87-9kcfx"] Jul 22 00:38:48.121672 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:48.121628 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afdd9a9a-c790-479c-a241-adaf589b2ba2" path="/var/lib/kubelet/pods/afdd9a9a-c790-479c-a241-adaf589b2ba2/volumes" Jul 22 00:38:54.749903 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:54.749836 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:54.848657 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:54.848616 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:57.722527 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:57.722491 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:38:57.823614 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:38:57.823574 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:00.326890 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:00.326850 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:00.428720 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:00.428676 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:03.924666 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:03.924626 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:04.025238 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:04.025072 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:07.124793 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:07.124745 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt" event={"ID":"6339b835-255f-4abf-90c3-8a26cee5522f","Type":"ContainerStarted","Data":"d73b30835ef4f710c28521f3cd0b67bb35daa8fc44fa1bdccc2aed727ee8f23f"} Jul 22 00:39:07.141289 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:07.141240 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8jlvt" podStartSLOduration=1.061877143 podStartE2EDuration="24.141224921s" totalImagesPullingTime="23.079347778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:39:07.139251642 +0000 UTC m=+755.465323844" watchObservedRunningTime="2026-07-22 00:39:07.141224921 +0000 UTC m=+755.467297123" Jul 22 00:39:10.917143 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:10.917105 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:11.016695 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:11.016658 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:14.136054 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:14.135993 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:14.235383 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:14.235354 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:17.129055 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:17.129014 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:17.230360 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:17.230334 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:26.022845 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:26.022802 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:26.259061 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:26.259022 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:41.292261 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:41.292226 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:41.376530 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:41.376501 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:41.430027 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:41.429999 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:42.076194 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.076161 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-zfzxl"] Jul 22 00:39:42.076698 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.076672 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afdd9a9a-c790-479c-a241-adaf589b2ba2" containerName="console" Jul 22 00:39:42.076698 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.076700 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="afdd9a9a-c790-479c-a241-adaf589b2ba2" containerName="console" Jul 22 00:39:42.076876 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.076800 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="afdd9a9a-c790-479c-a241-adaf589b2ba2" containerName="console" Jul 22 00:39:42.110228 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.110193 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:42.117354 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.117332 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-zfzxl"] Jul 22 00:39:42.117467 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.117440 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-zfzxl" Jul 22 00:39:42.120019 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.119985 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xxkp9\"" Jul 22 00:39:42.241418 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.241379 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whqsb\" (UniqueName: \"kubernetes.io/projected/7cebbfe5-9ba9-4d19-a674-e1f577a4ac62-kube-api-access-whqsb\") pod \"authorino-58cd955675-zfzxl\" (UID: \"7cebbfe5-9ba9-4d19-a674-e1f577a4ac62\") " pod="kuadrant-system/authorino-58cd955675-zfzxl" Jul 22 00:39:42.268720 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.268693 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-2ndnf"] Jul 22 00:39:42.272072 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.272057 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2ndnf" Jul 22 00:39:42.277433 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.277410 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2ndnf"] Jul 22 00:39:42.342126 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.342046 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whqsb\" (UniqueName: \"kubernetes.io/projected/7cebbfe5-9ba9-4d19-a674-e1f577a4ac62-kube-api-access-whqsb\") pod \"authorino-58cd955675-zfzxl\" (UID: \"7cebbfe5-9ba9-4d19-a674-e1f577a4ac62\") " pod="kuadrant-system/authorino-58cd955675-zfzxl" Jul 22 00:39:42.350003 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.349978 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whqsb\" (UniqueName: \"kubernetes.io/projected/7cebbfe5-9ba9-4d19-a674-e1f577a4ac62-kube-api-access-whqsb\") pod \"authorino-58cd955675-zfzxl\" (UID: \"7cebbfe5-9ba9-4d19-a674-e1f577a4ac62\") " pod="kuadrant-system/authorino-58cd955675-zfzxl" Jul 22 00:39:42.427568 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.427541 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-zfzxl" Jul 22 00:39:42.443199 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.443146 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6g6x\" (UniqueName: \"kubernetes.io/projected/2800903f-8136-4cab-b5ef-463a26a8bdf6-kube-api-access-b6g6x\") pod \"authorino-75d556bd77-2ndnf\" (UID: \"2800903f-8136-4cab-b5ef-463a26a8bdf6\") " pod="kuadrant-system/authorino-75d556bd77-2ndnf" Jul 22 00:39:42.543930 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.543902 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6g6x\" (UniqueName: \"kubernetes.io/projected/2800903f-8136-4cab-b5ef-463a26a8bdf6-kube-api-access-b6g6x\") pod \"authorino-75d556bd77-2ndnf\" (UID: \"2800903f-8136-4cab-b5ef-463a26a8bdf6\") " pod="kuadrant-system/authorino-75d556bd77-2ndnf" Jul 22 00:39:42.552577 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.552555 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-zfzxl"] Jul 22 00:39:42.554172 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.554146 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6g6x\" (UniqueName: \"kubernetes.io/projected/2800903f-8136-4cab-b5ef-463a26a8bdf6-kube-api-access-b6g6x\") pod \"authorino-75d556bd77-2ndnf\" (UID: \"2800903f-8136-4cab-b5ef-463a26a8bdf6\") " pod="kuadrant-system/authorino-75d556bd77-2ndnf" Jul 22 00:39:42.554503 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:39:42.554477 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cebbfe5_9ba9_4d19_a674_e1f577a4ac62.slice/crio-86e27657c16b21ad0c7e8954d1cca741576c16f50a71bd8fec69cfe55c98030c WatchSource:0}: Error finding container 86e27657c16b21ad0c7e8954d1cca741576c16f50a71bd8fec69cfe55c98030c: Status 404 returned error can't find the container with id 86e27657c16b21ad0c7e8954d1cca741576c16f50a71bd8fec69cfe55c98030c Jul 22 00:39:42.582319 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.582291 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2ndnf" Jul 22 00:39:42.704861 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.704834 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2ndnf"] Jul 22 00:39:42.707035 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:39:42.707007 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2800903f_8136_4cab_b5ef_463a26a8bdf6.slice/crio-77a79e2edb4d5a1ab58559d18f08258594e9294f35517d2bc5a56612b4599620 WatchSource:0}: Error finding container 77a79e2edb4d5a1ab58559d18f08258594e9294f35517d2bc5a56612b4599620: Status 404 returned error can't find the container with id 77a79e2edb4d5a1ab58559d18f08258594e9294f35517d2bc5a56612b4599620 Jul 22 00:39:42.765316 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:42.765290 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:43.297416 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:43.297385 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2ndnf" event={"ID":"2800903f-8136-4cab-b5ef-463a26a8bdf6","Type":"ContainerStarted","Data":"77a79e2edb4d5a1ab58559d18f08258594e9294f35517d2bc5a56612b4599620"} Jul 22 00:39:43.298542 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:43.298521 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-zfzxl" event={"ID":"7cebbfe5-9ba9-4d19-a674-e1f577a4ac62","Type":"ContainerStarted","Data":"86e27657c16b21ad0c7e8954d1cca741576c16f50a71bd8fec69cfe55c98030c"} Jul 22 00:39:44.877006 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:44.876967 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:47.324644 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:47.324605 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2ndnf" event={"ID":"2800903f-8136-4cab-b5ef-463a26a8bdf6","Type":"ContainerStarted","Data":"5e26025d0f59d95b04b097528575fe0eeacf4b90142dbf1e350b0be401cc82f8"} Jul 22 00:39:47.326236 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:47.326209 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-zfzxl" event={"ID":"7cebbfe5-9ba9-4d19-a674-e1f577a4ac62","Type":"ContainerStarted","Data":"988f357110045c6fa62eceae7d6b50b9aa32d202c492752a6dff94774559c15f"} Jul 22 00:39:47.338876 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:47.338832 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-2ndnf" podStartSLOduration=1.629804531 podStartE2EDuration="5.338817757s" totalImagesPullingTime="3.709013226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:39:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:39:47.338168745 +0000 UTC m=+795.664240964" watchObservedRunningTime="2026-07-22 00:39:47.338817757 +0000 UTC m=+795.664889959" Jul 22 00:39:47.351416 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:47.351368 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-zfzxl" podStartSLOduration=1.478717447 podStartE2EDuration="5.351357772s" totalImagesPullingTime="3.872640325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:39:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:39:47.350836733 +0000 UTC m=+795.676908938" watchObservedRunningTime="2026-07-22 00:39:47.351357772 +0000 UTC m=+795.677429974" Jul 22 00:39:47.377724 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:47.377695 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-zfzxl"] Jul 22 00:39:49.335486 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:49.335434 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-zfzxl" podUID="7cebbfe5-9ba9-4d19-a674-e1f577a4ac62" containerName="authorino" containerID="cri-o://988f357110045c6fa62eceae7d6b50b9aa32d202c492752a6dff94774559c15f" gracePeriod=30 Jul 22 00:39:49.579067 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:49.579044 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-zfzxl" Jul 22 00:39:49.598143 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:49.598084 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whqsb\" (UniqueName: \"kubernetes.io/projected/7cebbfe5-9ba9-4d19-a674-e1f577a4ac62-kube-api-access-whqsb\") pod \"7cebbfe5-9ba9-4d19-a674-e1f577a4ac62\" (UID: \"7cebbfe5-9ba9-4d19-a674-e1f577a4ac62\") " Jul 22 00:39:49.600188 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:49.600158 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cebbfe5-9ba9-4d19-a674-e1f577a4ac62-kube-api-access-whqsb" (OuterVolumeSpecName: "kube-api-access-whqsb") pod "7cebbfe5-9ba9-4d19-a674-e1f577a4ac62" (UID: "7cebbfe5-9ba9-4d19-a674-e1f577a4ac62"). InnerVolumeSpecName "kube-api-access-whqsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:39:49.699060 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:49.699037 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whqsb\" (UniqueName: \"kubernetes.io/projected/7cebbfe5-9ba9-4d19-a674-e1f577a4ac62-kube-api-access-whqsb\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:39:50.342209 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:50.342179 2567 generic.go:356] "Generic (PLEG): container finished" podID="7cebbfe5-9ba9-4d19-a674-e1f577a4ac62" containerID="988f357110045c6fa62eceae7d6b50b9aa32d202c492752a6dff94774559c15f" exitCode=0 Jul 22 00:39:50.342569 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:50.342214 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-zfzxl" event={"ID":"7cebbfe5-9ba9-4d19-a674-e1f577a4ac62","Type":"ContainerDied","Data":"988f357110045c6fa62eceae7d6b50b9aa32d202c492752a6dff94774559c15f"} Jul 22 00:39:50.342569 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:50.342222 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-zfzxl" Jul 22 00:39:50.342569 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:50.342239 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-zfzxl" event={"ID":"7cebbfe5-9ba9-4d19-a674-e1f577a4ac62","Type":"ContainerDied","Data":"86e27657c16b21ad0c7e8954d1cca741576c16f50a71bd8fec69cfe55c98030c"} Jul 22 00:39:50.342569 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:50.342256 2567 scope.go:117] "RemoveContainer" containerID="988f357110045c6fa62eceae7d6b50b9aa32d202c492752a6dff94774559c15f" Jul 22 00:39:50.350433 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:50.350413 2567 scope.go:117] "RemoveContainer" containerID="988f357110045c6fa62eceae7d6b50b9aa32d202c492752a6dff94774559c15f" Jul 22 00:39:50.350690 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:39:50.350668 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"988f357110045c6fa62eceae7d6b50b9aa32d202c492752a6dff94774559c15f\": container with ID starting with 988f357110045c6fa62eceae7d6b50b9aa32d202c492752a6dff94774559c15f not found: ID does not exist" containerID="988f357110045c6fa62eceae7d6b50b9aa32d202c492752a6dff94774559c15f" Jul 22 00:39:50.350747 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:50.350699 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988f357110045c6fa62eceae7d6b50b9aa32d202c492752a6dff94774559c15f"} err="failed to get container status \"988f357110045c6fa62eceae7d6b50b9aa32d202c492752a6dff94774559c15f\": rpc error: code = NotFound desc = could not find container \"988f357110045c6fa62eceae7d6b50b9aa32d202c492752a6dff94774559c15f\": container with ID starting with 988f357110045c6fa62eceae7d6b50b9aa32d202c492752a6dff94774559c15f not found: ID does not exist" Jul 22 00:39:50.357015 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:50.356991 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-zfzxl"] Jul 22 00:39:50.362844 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:50.362822 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-zfzxl"] Jul 22 00:39:52.118655 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:52.118609 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cebbfe5-9ba9-4d19-a674-e1f577a4ac62" path="/var/lib/kubelet/pods/7cebbfe5-9ba9-4d19-a674-e1f577a4ac62/volumes" Jul 22 00:39:59.144303 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:59.144259 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:59.241613 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:59.241579 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:39:59.575552 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:39:59.575518 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:14.582700 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:14.582661 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:14.806989 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:14.806949 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qppv8"] Jul 22 00:40:14.807316 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:14.807303 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cebbfe5-9ba9-4d19-a674-e1f577a4ac62" containerName="authorino" Jul 22 00:40:14.807362 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:14.807317 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cebbfe5-9ba9-4d19-a674-e1f577a4ac62" containerName="authorino" Jul 22 00:40:14.807398 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:14.807390 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cebbfe5-9ba9-4d19-a674-e1f577a4ac62" containerName="authorino" Jul 22 00:40:14.810618 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:14.810599 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-qppv8" Jul 22 00:40:14.817233 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:14.817200 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qppv8"] Jul 22 00:40:14.894539 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:14.894441 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lppdp\" (UniqueName: \"kubernetes.io/projected/79ed53b2-7837-4d77-a718-e36a3305e573-kube-api-access-lppdp\") pod \"authorino-5d4b67c446-qppv8\" (UID: \"79ed53b2-7837-4d77-a718-e36a3305e573\") " pod="kuadrant-system/authorino-5d4b67c446-qppv8" Jul 22 00:40:14.934711 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:14.934674 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:14.995262 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:14.995219 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lppdp\" (UniqueName: \"kubernetes.io/projected/79ed53b2-7837-4d77-a718-e36a3305e573-kube-api-access-lppdp\") pod \"authorino-5d4b67c446-qppv8\" (UID: \"79ed53b2-7837-4d77-a718-e36a3305e573\") " pod="kuadrant-system/authorino-5d4b67c446-qppv8" Jul 22 00:40:15.003335 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.003293 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lppdp\" (UniqueName: \"kubernetes.io/projected/79ed53b2-7837-4d77-a718-e36a3305e573-kube-api-access-lppdp\") pod \"authorino-5d4b67c446-qppv8\" (UID: \"79ed53b2-7837-4d77-a718-e36a3305e573\") " pod="kuadrant-system/authorino-5d4b67c446-qppv8" Jul 22 00:40:15.039818 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.039766 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qppv8"] Jul 22 00:40:15.040071 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.040049 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-qppv8" Jul 22 00:40:15.065310 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.065272 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c7b95f47d-vhf4v"] Jul 22 00:40:15.068912 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.068890 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c7b95f47d-vhf4v" Jul 22 00:40:15.074406 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.074378 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c7b95f47d-vhf4v"] Jul 22 00:40:15.142166 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.142134 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c7b95f47d-vhf4v"] Jul 22 00:40:15.142442 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:40:15.142423 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-bwhn2], unattached volumes=[], failed to process volumes=[kube-api-access-bwhn2]: context canceled" pod="kuadrant-system/authorino-c7b95f47d-vhf4v" podUID="2f810f6e-d673-4bbc-b622-8c4a94461d0b" Jul 22 00:40:15.167261 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.167188 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d7f8b89d-66pwl"] Jul 22 00:40:15.170460 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.170441 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d7f8b89d-66pwl" Jul 22 00:40:15.173393 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.173366 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 22 00:40:15.178298 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.178277 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d7f8b89d-66pwl"] Jul 22 00:40:15.186723 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.186658 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qppv8"] Jul 22 00:40:15.190473 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:40:15.190449 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79ed53b2_7837_4d77_a718_e36a3305e573.slice/crio-1f537ca88d7fba5d1877e24c9c95cc8004dac54e65ef8abb9e59586fda351126 WatchSource:0}: Error finding container 1f537ca88d7fba5d1877e24c9c95cc8004dac54e65ef8abb9e59586fda351126: Status 404 returned error can't find the container with id 1f537ca88d7fba5d1877e24c9c95cc8004dac54e65ef8abb9e59586fda351126 Jul 22 00:40:15.196683 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.196646 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwhn2\" (UniqueName: \"kubernetes.io/projected/2f810f6e-d673-4bbc-b622-8c4a94461d0b-kube-api-access-bwhn2\") pod \"authorino-c7b95f47d-vhf4v\" (UID: \"2f810f6e-d673-4bbc-b622-8c4a94461d0b\") " pod="kuadrant-system/authorino-c7b95f47d-vhf4v" Jul 22 00:40:15.297342 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.297305 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwhn2\" (UniqueName: \"kubernetes.io/projected/2f810f6e-d673-4bbc-b622-8c4a94461d0b-kube-api-access-bwhn2\") pod \"authorino-c7b95f47d-vhf4v\" (UID: \"2f810f6e-d673-4bbc-b622-8c4a94461d0b\") " pod="kuadrant-system/authorino-c7b95f47d-vhf4v" Jul 22 00:40:15.297342 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.297356 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2grhs\" (UniqueName: \"kubernetes.io/projected/0755509d-5f71-4838-b096-e66d16d861a1-kube-api-access-2grhs\") pod \"authorino-75d7f8b89d-66pwl\" (UID: \"0755509d-5f71-4838-b096-e66d16d861a1\") " pod="kuadrant-system/authorino-75d7f8b89d-66pwl" Jul 22 00:40:15.297584 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.297397 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0755509d-5f71-4838-b096-e66d16d861a1-tls-cert\") pod \"authorino-75d7f8b89d-66pwl\" (UID: \"0755509d-5f71-4838-b096-e66d16d861a1\") " pod="kuadrant-system/authorino-75d7f8b89d-66pwl" Jul 22 00:40:15.304859 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.304837 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwhn2\" (UniqueName: \"kubernetes.io/projected/2f810f6e-d673-4bbc-b622-8c4a94461d0b-kube-api-access-bwhn2\") pod \"authorino-c7b95f47d-vhf4v\" (UID: \"2f810f6e-d673-4bbc-b622-8c4a94461d0b\") " pod="kuadrant-system/authorino-c7b95f47d-vhf4v" Jul 22 00:40:15.398558 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.398520 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0755509d-5f71-4838-b096-e66d16d861a1-tls-cert\") pod \"authorino-75d7f8b89d-66pwl\" (UID: \"0755509d-5f71-4838-b096-e66d16d861a1\") " pod="kuadrant-system/authorino-75d7f8b89d-66pwl" Jul 22 00:40:15.398765 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.398624 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2grhs\" (UniqueName: \"kubernetes.io/projected/0755509d-5f71-4838-b096-e66d16d861a1-kube-api-access-2grhs\") pod \"authorino-75d7f8b89d-66pwl\" (UID: \"0755509d-5f71-4838-b096-e66d16d861a1\") " pod="kuadrant-system/authorino-75d7f8b89d-66pwl" Jul 22 00:40:15.400958 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.400937 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0755509d-5f71-4838-b096-e66d16d861a1-tls-cert\") pod \"authorino-75d7f8b89d-66pwl\" (UID: \"0755509d-5f71-4838-b096-e66d16d861a1\") " pod="kuadrant-system/authorino-75d7f8b89d-66pwl" Jul 22 00:40:15.405958 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.405937 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2grhs\" (UniqueName: \"kubernetes.io/projected/0755509d-5f71-4838-b096-e66d16d861a1-kube-api-access-2grhs\") pod \"authorino-75d7f8b89d-66pwl\" (UID: \"0755509d-5f71-4838-b096-e66d16d861a1\") " pod="kuadrant-system/authorino-75d7f8b89d-66pwl" Jul 22 00:40:15.460159 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.460074 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-qppv8" event={"ID":"79ed53b2-7837-4d77-a718-e36a3305e573","Type":"ContainerStarted","Data":"1f537ca88d7fba5d1877e24c9c95cc8004dac54e65ef8abb9e59586fda351126"} Jul 22 00:40:15.460159 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.460103 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c7b95f47d-vhf4v" Jul 22 00:40:15.468323 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.468303 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c7b95f47d-vhf4v" Jul 22 00:40:15.481095 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.481070 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d7f8b89d-66pwl" Jul 22 00:40:15.600169 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.600137 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwhn2\" (UniqueName: \"kubernetes.io/projected/2f810f6e-d673-4bbc-b622-8c4a94461d0b-kube-api-access-bwhn2\") pod \"2f810f6e-d673-4bbc-b622-8c4a94461d0b\" (UID: \"2f810f6e-d673-4bbc-b622-8c4a94461d0b\") " Jul 22 00:40:15.602110 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.602084 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f810f6e-d673-4bbc-b622-8c4a94461d0b-kube-api-access-bwhn2" (OuterVolumeSpecName: "kube-api-access-bwhn2") pod "2f810f6e-d673-4bbc-b622-8c4a94461d0b" (UID: "2f810f6e-d673-4bbc-b622-8c4a94461d0b"). InnerVolumeSpecName "kube-api-access-bwhn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:40:15.614699 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.614673 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d7f8b89d-66pwl"] Jul 22 00:40:15.616336 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:40:15.616314 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0755509d_5f71_4838_b096_e66d16d861a1.slice/crio-13356dbb29536f84596fbbe8fe9c61aa90afd14d04c54895dd504df7d5129c0e WatchSource:0}: Error finding container 13356dbb29536f84596fbbe8fe9c61aa90afd14d04c54895dd504df7d5129c0e: Status 404 returned error can't find the container with id 13356dbb29536f84596fbbe8fe9c61aa90afd14d04c54895dd504df7d5129c0e Jul 22 00:40:15.701180 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:15.701128 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwhn2\" (UniqueName: \"kubernetes.io/projected/2f810f6e-d673-4bbc-b622-8c4a94461d0b-kube-api-access-bwhn2\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:40:16.466641 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.466596 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d7f8b89d-66pwl" event={"ID":"0755509d-5f71-4838-b096-e66d16d861a1","Type":"ContainerStarted","Data":"85ac59df5868b312f87fb5d85bf080d1344b492f325d90a417155fea16c89654"} Jul 22 00:40:16.466834 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.466649 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d7f8b89d-66pwl" event={"ID":"0755509d-5f71-4838-b096-e66d16d861a1","Type":"ContainerStarted","Data":"13356dbb29536f84596fbbe8fe9c61aa90afd14d04c54895dd504df7d5129c0e"} Jul 22 00:40:16.468151 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.468112 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-qppv8" event={"ID":"79ed53b2-7837-4d77-a718-e36a3305e573","Type":"ContainerStarted","Data":"d40281488b92eee6762033240a31c87307fcf4b69dea65280d9b2207bec11711"} Jul 22 00:40:16.468151 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.468127 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c7b95f47d-vhf4v" Jul 22 00:40:16.468329 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.468176 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-qppv8" podUID="79ed53b2-7837-4d77-a718-e36a3305e573" containerName="authorino" containerID="cri-o://d40281488b92eee6762033240a31c87307fcf4b69dea65280d9b2207bec11711" gracePeriod=30 Jul 22 00:40:16.483208 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.483146 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d7f8b89d-66pwl" podStartSLOduration=0.898927015 podStartE2EDuration="1.483130956s" totalImagesPullingTime="584.203941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:40:16.481847736 +0000 UTC m=+824.807919935" watchObservedRunningTime="2026-07-22 00:40:16.483130956 +0000 UTC m=+824.809203171" Jul 22 00:40:16.506705 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.506671 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c7b95f47d-vhf4v"] Jul 22 00:40:16.512666 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.512633 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2ndnf"] Jul 22 00:40:16.512904 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.512878 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-2ndnf" podUID="2800903f-8136-4cab-b5ef-463a26a8bdf6" containerName="authorino" containerID="cri-o://5e26025d0f59d95b04b097528575fe0eeacf4b90142dbf1e350b0be401cc82f8" gracePeriod=30 Jul 22 00:40:16.514663 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.514637 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c7b95f47d-vhf4v"] Jul 22 00:40:16.529375 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.529328 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-qppv8" podStartSLOduration=2.149366213 podStartE2EDuration="2.529311698s" totalImagesPullingTime="379.945485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:40:16.528262501 +0000 UTC m=+824.854334703" watchObservedRunningTime="2026-07-22 00:40:16.529311698 +0000 UTC m=+824.855383900" Jul 22 00:40:16.761836 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.761807 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-qppv8" Jul 22 00:40:16.766968 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.766943 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2ndnf" Jul 22 00:40:16.809844 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.809749 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lppdp\" (UniqueName: \"kubernetes.io/projected/79ed53b2-7837-4d77-a718-e36a3305e573-kube-api-access-lppdp\") pod \"79ed53b2-7837-4d77-a718-e36a3305e573\" (UID: \"79ed53b2-7837-4d77-a718-e36a3305e573\") " Jul 22 00:40:16.809974 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.809893 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6g6x\" (UniqueName: \"kubernetes.io/projected/2800903f-8136-4cab-b5ef-463a26a8bdf6-kube-api-access-b6g6x\") pod \"2800903f-8136-4cab-b5ef-463a26a8bdf6\" (UID: \"2800903f-8136-4cab-b5ef-463a26a8bdf6\") " Jul 22 00:40:16.811719 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.811689 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ed53b2-7837-4d77-a718-e36a3305e573-kube-api-access-lppdp" (OuterVolumeSpecName: "kube-api-access-lppdp") pod "79ed53b2-7837-4d77-a718-e36a3305e573" (UID: "79ed53b2-7837-4d77-a718-e36a3305e573"). InnerVolumeSpecName "kube-api-access-lppdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:40:16.811848 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.811810 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2800903f-8136-4cab-b5ef-463a26a8bdf6-kube-api-access-b6g6x" (OuterVolumeSpecName: "kube-api-access-b6g6x") pod "2800903f-8136-4cab-b5ef-463a26a8bdf6" (UID: "2800903f-8136-4cab-b5ef-463a26a8bdf6"). InnerVolumeSpecName "kube-api-access-b6g6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:40:16.911076 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.911034 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lppdp\" (UniqueName: \"kubernetes.io/projected/79ed53b2-7837-4d77-a718-e36a3305e573-kube-api-access-lppdp\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:40:16.911076 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:16.911072 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6g6x\" (UniqueName: \"kubernetes.io/projected/2800903f-8136-4cab-b5ef-463a26a8bdf6-kube-api-access-b6g6x\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:40:17.475083 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.475049 2567 generic.go:356] "Generic (PLEG): container finished" podID="2800903f-8136-4cab-b5ef-463a26a8bdf6" containerID="5e26025d0f59d95b04b097528575fe0eeacf4b90142dbf1e350b0be401cc82f8" exitCode=0 Jul 22 00:40:17.475246 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.475092 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-2ndnf" Jul 22 00:40:17.475246 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.475140 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2ndnf" event={"ID":"2800903f-8136-4cab-b5ef-463a26a8bdf6","Type":"ContainerDied","Data":"5e26025d0f59d95b04b097528575fe0eeacf4b90142dbf1e350b0be401cc82f8"} Jul 22 00:40:17.475246 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.475194 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-2ndnf" event={"ID":"2800903f-8136-4cab-b5ef-463a26a8bdf6","Type":"ContainerDied","Data":"77a79e2edb4d5a1ab58559d18f08258594e9294f35517d2bc5a56612b4599620"} Jul 22 00:40:17.475246 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.475214 2567 scope.go:117] "RemoveContainer" containerID="5e26025d0f59d95b04b097528575fe0eeacf4b90142dbf1e350b0be401cc82f8" Jul 22 00:40:17.476538 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.476516 2567 generic.go:356] "Generic (PLEG): container finished" podID="79ed53b2-7837-4d77-a718-e36a3305e573" containerID="d40281488b92eee6762033240a31c87307fcf4b69dea65280d9b2207bec11711" exitCode=0 Jul 22 00:40:17.476611 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.476567 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-qppv8" Jul 22 00:40:17.476662 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.476620 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-qppv8" event={"ID":"79ed53b2-7837-4d77-a718-e36a3305e573","Type":"ContainerDied","Data":"d40281488b92eee6762033240a31c87307fcf4b69dea65280d9b2207bec11711"} Jul 22 00:40:17.476662 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.476653 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-qppv8" event={"ID":"79ed53b2-7837-4d77-a718-e36a3305e573","Type":"ContainerDied","Data":"1f537ca88d7fba5d1877e24c9c95cc8004dac54e65ef8abb9e59586fda351126"} Jul 22 00:40:17.483815 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.483774 2567 scope.go:117] "RemoveContainer" containerID="5e26025d0f59d95b04b097528575fe0eeacf4b90142dbf1e350b0be401cc82f8" Jul 22 00:40:17.484100 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:40:17.484076 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e26025d0f59d95b04b097528575fe0eeacf4b90142dbf1e350b0be401cc82f8\": container with ID starting with 5e26025d0f59d95b04b097528575fe0eeacf4b90142dbf1e350b0be401cc82f8 not found: ID does not exist" containerID="5e26025d0f59d95b04b097528575fe0eeacf4b90142dbf1e350b0be401cc82f8" Jul 22 00:40:17.484176 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.484110 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e26025d0f59d95b04b097528575fe0eeacf4b90142dbf1e350b0be401cc82f8"} err="failed to get container status \"5e26025d0f59d95b04b097528575fe0eeacf4b90142dbf1e350b0be401cc82f8\": rpc error: code = NotFound desc = could not find container \"5e26025d0f59d95b04b097528575fe0eeacf4b90142dbf1e350b0be401cc82f8\": container with ID starting with 5e26025d0f59d95b04b097528575fe0eeacf4b90142dbf1e350b0be401cc82f8 not found: ID does not exist" Jul 22 00:40:17.484176 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.484132 2567 scope.go:117] "RemoveContainer" containerID="d40281488b92eee6762033240a31c87307fcf4b69dea65280d9b2207bec11711" Jul 22 00:40:17.492625 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.492608 2567 scope.go:117] "RemoveContainer" containerID="d40281488b92eee6762033240a31c87307fcf4b69dea65280d9b2207bec11711" Jul 22 00:40:17.492953 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:40:17.492930 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d40281488b92eee6762033240a31c87307fcf4b69dea65280d9b2207bec11711\": container with ID starting with d40281488b92eee6762033240a31c87307fcf4b69dea65280d9b2207bec11711 not found: ID does not exist" containerID="d40281488b92eee6762033240a31c87307fcf4b69dea65280d9b2207bec11711" Jul 22 00:40:17.493036 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.492965 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d40281488b92eee6762033240a31c87307fcf4b69dea65280d9b2207bec11711"} err="failed to get container status \"d40281488b92eee6762033240a31c87307fcf4b69dea65280d9b2207bec11711\": rpc error: code = NotFound desc = could not find container \"d40281488b92eee6762033240a31c87307fcf4b69dea65280d9b2207bec11711\": container with ID starting with d40281488b92eee6762033240a31c87307fcf4b69dea65280d9b2207bec11711 not found: ID does not exist" Jul 22 00:40:17.500331 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.500306 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2ndnf"] Jul 22 00:40:17.505405 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.505370 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-2ndnf"] Jul 22 00:40:17.515699 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.515672 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qppv8"] Jul 22 00:40:17.520240 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:17.520215 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qppv8"] Jul 22 00:40:18.117885 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:18.117847 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2800903f-8136-4cab-b5ef-463a26a8bdf6" path="/var/lib/kubelet/pods/2800903f-8136-4cab-b5ef-463a26a8bdf6/volumes" Jul 22 00:40:18.118263 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:18.118186 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f810f6e-d673-4bbc-b622-8c4a94461d0b" path="/var/lib/kubelet/pods/2f810f6e-d673-4bbc-b622-8c4a94461d0b/volumes" Jul 22 00:40:18.118376 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:18.118363 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79ed53b2-7837-4d77-a718-e36a3305e573" path="/var/lib/kubelet/pods/79ed53b2-7837-4d77-a718-e36a3305e573/volumes" Jul 22 00:40:21.097506 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:21.097471 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:21.263432 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:21.263399 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:21.319396 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:21.319361 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:21.455422 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:21.455336 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:24.566618 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:24.566584 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:24.661370 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:24.661342 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:32.062205 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:32.062168 2567 scope.go:117] "RemoveContainer" containerID="bb3b94fef7915d2ade34904d543d7784edcc5e3839f4409d52e85c0198b4109e" Jul 22 00:40:32.152589 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:32.152559 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:32.254731 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:32.254691 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:56.206713 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:56.206675 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:56.306546 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:56.306513 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:56.800692 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:56.800655 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:57.255316 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.255165 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:57.349643 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.349588 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:40:57.353908 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.353880 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-xphnd"] Jul 22 00:40:57.354239 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.354226 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ed53b2-7837-4d77-a718-e36a3305e573" containerName="authorino" Jul 22 00:40:57.354282 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.354241 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ed53b2-7837-4d77-a718-e36a3305e573" containerName="authorino" Jul 22 00:40:57.354282 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.354262 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2800903f-8136-4cab-b5ef-463a26a8bdf6" containerName="authorino" Jul 22 00:40:57.354282 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.354268 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2800903f-8136-4cab-b5ef-463a26a8bdf6" containerName="authorino" Jul 22 00:40:57.354389 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.354323 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2800903f-8136-4cab-b5ef-463a26a8bdf6" containerName="authorino" Jul 22 00:40:57.354389 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.354332 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ed53b2-7837-4d77-a718-e36a3305e573" containerName="authorino" Jul 22 00:40:57.358741 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.358725 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-xphnd" Jul 22 00:40:57.363855 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.363832 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bdj8v\"" Jul 22 00:40:57.363855 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.363832 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 22 00:40:57.371157 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.371130 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-xphnd"] Jul 22 00:40:57.375289 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.375265 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-xx7xj"] Jul 22 00:40:57.384253 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.384228 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-xx7xj" Jul 22 00:40:57.389806 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.389761 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-xx7xj"] Jul 22 00:40:57.461798 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.461738 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbrwd\" (UniqueName: \"kubernetes.io/projected/7fa3fb06-305b-4d41-91cc-ab7a360ef18f-kube-api-access-cbrwd\") pod \"payload-pre-processing-7677b774bb-xphnd\" (UID: \"7fa3fb06-305b-4d41-91cc-ab7a360ef18f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xphnd" Jul 22 00:40:57.461962 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.461830 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b48eb0-702c-4da5-992d-07def03753ca-plugins-config-volume\") pod \"payload-processing-5bf47cd988-xx7xj\" (UID: \"69b48eb0-702c-4da5-992d-07def03753ca\") " pod="openshift-ingress/payload-processing-5bf47cd988-xx7xj" Jul 22 00:40:57.461962 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.461858 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa3fb06-305b-4d41-91cc-ab7a360ef18f-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-xphnd\" (UID: \"7fa3fb06-305b-4d41-91cc-ab7a360ef18f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xphnd" Jul 22 00:40:57.461962 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.461879 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8vgw\" (UniqueName: \"kubernetes.io/projected/69b48eb0-702c-4da5-992d-07def03753ca-kube-api-access-n8vgw\") pod \"payload-processing-5bf47cd988-xx7xj\" (UID: \"69b48eb0-702c-4da5-992d-07def03753ca\") " pod="openshift-ingress/payload-processing-5bf47cd988-xx7xj" Jul 22 00:40:57.562558 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.562518 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbrwd\" (UniqueName: \"kubernetes.io/projected/7fa3fb06-305b-4d41-91cc-ab7a360ef18f-kube-api-access-cbrwd\") pod \"payload-pre-processing-7677b774bb-xphnd\" (UID: \"7fa3fb06-305b-4d41-91cc-ab7a360ef18f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xphnd" Jul 22 00:40:57.562768 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.562571 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b48eb0-702c-4da5-992d-07def03753ca-plugins-config-volume\") pod \"payload-processing-5bf47cd988-xx7xj\" (UID: \"69b48eb0-702c-4da5-992d-07def03753ca\") " pod="openshift-ingress/payload-processing-5bf47cd988-xx7xj" Jul 22 00:40:57.562768 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.562591 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa3fb06-305b-4d41-91cc-ab7a360ef18f-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-xphnd\" (UID: \"7fa3fb06-305b-4d41-91cc-ab7a360ef18f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xphnd" Jul 22 00:40:57.562768 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.562613 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8vgw\" (UniqueName: \"kubernetes.io/projected/69b48eb0-702c-4da5-992d-07def03753ca-kube-api-access-n8vgw\") pod \"payload-processing-5bf47cd988-xx7xj\" (UID: \"69b48eb0-702c-4da5-992d-07def03753ca\") " pod="openshift-ingress/payload-processing-5bf47cd988-xx7xj" Jul 22 00:40:57.563304 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.563282 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b48eb0-702c-4da5-992d-07def03753ca-plugins-config-volume\") pod \"payload-processing-5bf47cd988-xx7xj\" (UID: \"69b48eb0-702c-4da5-992d-07def03753ca\") " pod="openshift-ingress/payload-processing-5bf47cd988-xx7xj" Jul 22 00:40:57.563399 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.563380 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa3fb06-305b-4d41-91cc-ab7a360ef18f-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-xphnd\" (UID: \"7fa3fb06-305b-4d41-91cc-ab7a360ef18f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xphnd" Jul 22 00:40:57.570744 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.570715 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8vgw\" (UniqueName: \"kubernetes.io/projected/69b48eb0-702c-4da5-992d-07def03753ca-kube-api-access-n8vgw\") pod \"payload-processing-5bf47cd988-xx7xj\" (UID: \"69b48eb0-702c-4da5-992d-07def03753ca\") " pod="openshift-ingress/payload-processing-5bf47cd988-xx7xj" Jul 22 00:40:57.570867 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.570818 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbrwd\" (UniqueName: \"kubernetes.io/projected/7fa3fb06-305b-4d41-91cc-ab7a360ef18f-kube-api-access-cbrwd\") pod \"payload-pre-processing-7677b774bb-xphnd\" (UID: \"7fa3fb06-305b-4d41-91cc-ab7a360ef18f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xphnd" Jul 22 00:40:57.669495 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.669453 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-xphnd" Jul 22 00:40:57.696538 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.696508 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-xx7xj" Jul 22 00:40:57.855806 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.855765 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-xphnd"] Jul 22 00:40:57.857985 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:40:57.857947 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa3fb06_305b_4d41_91cc_ab7a360ef18f.slice/crio-d8a97891c0bf8b7c5712076d946c8dc84efa1ae9ff61e74bfddec18c36e7497c WatchSource:0}: Error finding container d8a97891c0bf8b7c5712076d946c8dc84efa1ae9ff61e74bfddec18c36e7497c: Status 404 returned error can't find the container with id d8a97891c0bf8b7c5712076d946c8dc84efa1ae9ff61e74bfddec18c36e7497c Jul 22 00:40:57.899902 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:57.899878 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-xx7xj"] Jul 22 00:40:57.901726 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:40:57.901687 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b48eb0_702c_4da5_992d_07def03753ca.slice/crio-17ceadfe5fe2768420e6eb18c539a45d600f24098d2f99bec01e74402ae1dadf WatchSource:0}: Error finding container 17ceadfe5fe2768420e6eb18c539a45d600f24098d2f99bec01e74402ae1dadf: Status 404 returned error can't find the container with id 17ceadfe5fe2768420e6eb18c539a45d600f24098d2f99bec01e74402ae1dadf Jul 22 00:40:58.673107 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:58.673067 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-xx7xj" event={"ID":"69b48eb0-702c-4da5-992d-07def03753ca","Type":"ContainerStarted","Data":"17ceadfe5fe2768420e6eb18c539a45d600f24098d2f99bec01e74402ae1dadf"} Jul 22 00:40:58.674460 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:58.674433 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-xphnd" event={"ID":"7fa3fb06-305b-4d41-91cc-ab7a360ef18f","Type":"ContainerStarted","Data":"d8a97891c0bf8b7c5712076d946c8dc84efa1ae9ff61e74bfddec18c36e7497c"} Jul 22 00:40:59.685131 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:59.684825 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-xx7xj" event={"ID":"69b48eb0-702c-4da5-992d-07def03753ca","Type":"ContainerStarted","Data":"9d009bdc91d881a0537aaa21269516ae95a9698ff3e1e53b0e44fed7e8d6a309"} Jul 22 00:40:59.685131 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:59.685019 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-xx7xj" Jul 22 00:40:59.701355 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:40:59.701297 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-xx7xj" podStartSLOduration=1.019238536 podStartE2EDuration="2.701278239s" totalImagesPullingTime="1.682039703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:40:59.699511251 +0000 UTC m=+868.025583466" watchObservedRunningTime="2026-07-22 00:40:59.701278239 +0000 UTC m=+868.027350444" Jul 22 00:41:00.691667 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:00.691629 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-xphnd" event={"ID":"7fa3fb06-305b-4d41-91cc-ab7a360ef18f","Type":"ContainerStarted","Data":"33768d971fa11d685e13e804d25d32ccf4f4bdf657c89db12f2e815eec256445"} Jul 22 00:41:00.707314 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:00.707246 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-xphnd" podStartSLOduration=1.511026344 podStartE2EDuration="3.707226671s" totalImagesPullingTime="2.196200327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:41:00.706420935 +0000 UTC m=+869.032493136" watchObservedRunningTime="2026-07-22 00:41:00.707226671 +0000 UTC m=+869.033298872" Jul 22 00:41:01.696851 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:01.696815 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-xphnd" Jul 22 00:41:07.487380 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:07.487313 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:41:07.587470 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:07.587429 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:41:10.693988 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:10.693948 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-xx7xj" Jul 22 00:41:12.703459 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:12.703428 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-xphnd" Jul 22 00:41:30.500336 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:30.500250 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:41:35.889131 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:35.889094 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj"] Jul 22 00:41:35.892515 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:35.892493 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:35.896144 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:35.896119 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 00:41:35.896144 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:35.896129 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 00:41:35.897432 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:35.897404 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jsfjh\"" Jul 22 00:41:35.897610 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:35.897503 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 22 00:41:35.901025 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:35.901007 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8268b328-5b2f-4c1d-bfd4-83a74c5df038-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:35.901124 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:35.901051 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzf5n\" (UniqueName: \"kubernetes.io/projected/8268b328-5b2f-4c1d-bfd4-83a74c5df038-kube-api-access-jzf5n\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:35.901124 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:35.901070 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8268b328-5b2f-4c1d-bfd4-83a74c5df038-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:35.901247 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:35.901121 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8268b328-5b2f-4c1d-bfd4-83a74c5df038-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:35.901247 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:35.901178 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8268b328-5b2f-4c1d-bfd4-83a74c5df038-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:35.901247 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:35.901241 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8268b328-5b2f-4c1d-bfd4-83a74c5df038-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:35.904571 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:35.904550 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj"] Jul 22 00:41:35.925052 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:35.925028 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:41:36.001733 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:36.001692 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8268b328-5b2f-4c1d-bfd4-83a74c5df038-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:36.001942 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:36.001758 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzf5n\" (UniqueName: \"kubernetes.io/projected/8268b328-5b2f-4c1d-bfd4-83a74c5df038-kube-api-access-jzf5n\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:36.001942 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:36.001812 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8268b328-5b2f-4c1d-bfd4-83a74c5df038-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:36.001942 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:36.001838 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8268b328-5b2f-4c1d-bfd4-83a74c5df038-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:36.001942 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:36.001876 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8268b328-5b2f-4c1d-bfd4-83a74c5df038-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:36.001942 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:36.001933 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8268b328-5b2f-4c1d-bfd4-83a74c5df038-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:36.002309 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:36.002277 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8268b328-5b2f-4c1d-bfd4-83a74c5df038-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:36.002309 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:36.002302 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8268b328-5b2f-4c1d-bfd4-83a74c5df038-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:36.002460 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:36.002362 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8268b328-5b2f-4c1d-bfd4-83a74c5df038-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:36.003998 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:36.003971 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8268b328-5b2f-4c1d-bfd4-83a74c5df038-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:36.004271 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:36.004254 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8268b328-5b2f-4c1d-bfd4-83a74c5df038-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:36.010110 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:36.010092 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzf5n\" (UniqueName: \"kubernetes.io/projected/8268b328-5b2f-4c1d-bfd4-83a74c5df038-kube-api-access-jzf5n\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vq6jj\" (UID: \"8268b328-5b2f-4c1d-bfd4-83a74c5df038\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:36.204433 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:36.204345 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:36.336683 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:36.336652 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj"] Jul 22 00:41:36.338088 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:41:36.338064 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8268b328_5b2f_4c1d_bfd4_83a74c5df038.slice/crio-b96593953c86a73cc1bbadc2cfa48592252ef57f19bbecb43d92b0e8013db205 WatchSource:0}: Error finding container b96593953c86a73cc1bbadc2cfa48592252ef57f19bbecb43d92b0e8013db205: Status 404 returned error can't find the container with id b96593953c86a73cc1bbadc2cfa48592252ef57f19bbecb43d92b0e8013db205 Jul 22 00:41:36.861400 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:36.861355 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" event={"ID":"8268b328-5b2f-4c1d-bfd4-83a74c5df038","Type":"ContainerStarted","Data":"b96593953c86a73cc1bbadc2cfa48592252ef57f19bbecb43d92b0e8013db205"} Jul 22 00:41:37.868093 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:37.868054 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" event={"ID":"8268b328-5b2f-4c1d-bfd4-83a74c5df038","Type":"ContainerStarted","Data":"64d5ea655cab14e9da3d5bb26cf24d1502677f4d75495ef1e3c7d4c2e2fd3b7a"} Jul 22 00:41:37.884935 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:37.884892 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" podStartSLOduration=1.797029625 podStartE2EDuration="2.884877077s" totalImagesPullingTime="1.087847452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:41:37.884472014 +0000 UTC m=+906.210544249" watchObservedRunningTime="2026-07-22 00:41:37.884877077 +0000 UTC m=+906.210949279" Jul 22 00:41:40.084372 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:40.084341 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:41:40.085411 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:40.085387 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:41:41.093944 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:41.093901 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:41:42.002562 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.002522 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8"] Jul 22 00:41:42.006718 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.006696 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.009035 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.009010 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 22 00:41:42.015022 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.014998 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8"] Jul 22 00:41:42.032965 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.032922 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:41:42.056215 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.056186 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e25da29-4761-441b-bb94-77fbc447537b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.056355 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.056219 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e25da29-4761-441b-bb94-77fbc447537b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.056355 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.056244 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e25da29-4761-441b-bb94-77fbc447537b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.056427 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.056347 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e25da29-4761-441b-bb94-77fbc447537b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.056427 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.056404 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e25da29-4761-441b-bb94-77fbc447537b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.056427 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.056420 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrd5g\" (UniqueName: \"kubernetes.io/projected/7e25da29-4761-441b-bb94-77fbc447537b-kube-api-access-rrd5g\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.157436 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.157397 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e25da29-4761-441b-bb94-77fbc447537b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.157436 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.157444 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrd5g\" (UniqueName: \"kubernetes.io/projected/7e25da29-4761-441b-bb94-77fbc447537b-kube-api-access-rrd5g\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.157951 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.157478 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e25da29-4761-441b-bb94-77fbc447537b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.157951 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.157496 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e25da29-4761-441b-bb94-77fbc447537b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.157951 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.157529 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e25da29-4761-441b-bb94-77fbc447537b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.157951 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.157814 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e25da29-4761-441b-bb94-77fbc447537b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.157951 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.157885 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e25da29-4761-441b-bb94-77fbc447537b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.158224 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.157981 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e25da29-4761-441b-bb94-77fbc447537b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.158224 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.158173 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e25da29-4761-441b-bb94-77fbc447537b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.159740 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.159710 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e25da29-4761-441b-bb94-77fbc447537b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.160049 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.160029 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e25da29-4761-441b-bb94-77fbc447537b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.164998 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.164976 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrd5g\" (UniqueName: \"kubernetes.io/projected/7e25da29-4761-441b-bb94-77fbc447537b-kube-api-access-rrd5g\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hrr8\" (UID: \"7e25da29-4761-441b-bb94-77fbc447537b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.318380 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.318346 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:42.453840 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.453815 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8"] Jul 22 00:41:42.456238 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:41:42.456207 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e25da29_4761_441b_bb94_77fbc447537b.slice/crio-c5a78c2ae721fc998ea3c53cf3444e5e7ff3ba2948cbb9c59b186ee2ef7bc32b WatchSource:0}: Error finding container c5a78c2ae721fc998ea3c53cf3444e5e7ff3ba2948cbb9c59b186ee2ef7bc32b: Status 404 returned error can't find the container with id c5a78c2ae721fc998ea3c53cf3444e5e7ff3ba2948cbb9c59b186ee2ef7bc32b Jul 22 00:41:42.457686 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.457670 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 00:41:42.893806 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.893697 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" event={"ID":"7e25da29-4761-441b-bb94-77fbc447537b","Type":"ContainerStarted","Data":"8ae4355643bee27980c208e95a130668a9159e021900d72393004765ff15b9e1"} Jul 22 00:41:42.893806 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.893752 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" event={"ID":"7e25da29-4761-441b-bb94-77fbc447537b","Type":"ContainerStarted","Data":"c5a78c2ae721fc998ea3c53cf3444e5e7ff3ba2948cbb9c59b186ee2ef7bc32b"} Jul 22 00:41:42.909762 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:42.909712 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" podStartSLOduration=1.726623016 podStartE2EDuration="1.90969513s" totalImagesPullingTime="183.072114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:41:42.908261238 +0000 UTC m=+911.234333465" watchObservedRunningTime="2026-07-22 00:41:42.90969513 +0000 UTC m=+911.235767332" Jul 22 00:41:46.205115 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:46.205089 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:46.205481 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:46.205138 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:46.217916 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:46.217888 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:46.923979 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:46.923941 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vq6jj" Jul 22 00:41:52.318613 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:52.318564 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:52.319158 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:52.318637 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:52.331675 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:52.331644 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:41:52.530179 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:52.530150 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:41:52.951798 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:41:52.951757 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hrr8" Jul 22 00:42:00.621302 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:00.621254 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:42:00.692914 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:00.692869 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:42:03.502873 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.502837 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs"] Jul 22 00:42:03.506499 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.506475 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.508897 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.508874 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 22 00:42:03.515757 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.515729 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs"] Jul 22 00:42:03.534577 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.534544 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:42:03.656049 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.656004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8effc54f-fa47-4e08-88ee-c49ea54855db-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.656227 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.656076 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8effc54f-fa47-4e08-88ee-c49ea54855db-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.656227 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.656103 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8effc54f-fa47-4e08-88ee-c49ea54855db-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.656227 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.656124 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8effc54f-fa47-4e08-88ee-c49ea54855db-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.656332 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.656255 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8effc54f-fa47-4e08-88ee-c49ea54855db-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.656332 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.656288 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-879xx\" (UniqueName: \"kubernetes.io/projected/8effc54f-fa47-4e08-88ee-c49ea54855db-kube-api-access-879xx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.757765 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.757664 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8effc54f-fa47-4e08-88ee-c49ea54855db-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.757765 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.757712 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8effc54f-fa47-4e08-88ee-c49ea54855db-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.758051 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.757812 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8effc54f-fa47-4e08-88ee-c49ea54855db-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.758051 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.757931 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-879xx\" (UniqueName: \"kubernetes.io/projected/8effc54f-fa47-4e08-88ee-c49ea54855db-kube-api-access-879xx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.758051 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.757998 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8effc54f-fa47-4e08-88ee-c49ea54855db-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.758051 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.758030 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8effc54f-fa47-4e08-88ee-c49ea54855db-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.758250 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.758111 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8effc54f-fa47-4e08-88ee-c49ea54855db-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.758250 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.758133 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8effc54f-fa47-4e08-88ee-c49ea54855db-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.758325 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.758283 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8effc54f-fa47-4e08-88ee-c49ea54855db-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.760108 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.760077 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8effc54f-fa47-4e08-88ee-c49ea54855db-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.760596 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.760577 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8effc54f-fa47-4e08-88ee-c49ea54855db-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.765938 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.765913 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-879xx\" (UniqueName: \"kubernetes.io/projected/8effc54f-fa47-4e08-88ee-c49ea54855db-kube-api-access-879xx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4pcs\" (UID: \"8effc54f-fa47-4e08-88ee-c49ea54855db\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.818033 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.817974 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:03.959007 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.958978 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs"] Jul 22 00:42:03.961250 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:42:03.961220 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8effc54f_fa47_4e08_88ee_c49ea54855db.slice/crio-c34ccdc4636faaa713c21de88a652c388aa16750f767a9ff3644c730757f1155 WatchSource:0}: Error finding container c34ccdc4636faaa713c21de88a652c388aa16750f767a9ff3644c730757f1155: Status 404 returned error can't find the container with id c34ccdc4636faaa713c21de88a652c388aa16750f767a9ff3644c730757f1155 Jul 22 00:42:03.993151 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:03.993110 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" event={"ID":"8effc54f-fa47-4e08-88ee-c49ea54855db","Type":"ContainerStarted","Data":"c34ccdc4636faaa713c21de88a652c388aa16750f767a9ff3644c730757f1155"} Jul 22 00:42:04.999744 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:04.999709 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" event={"ID":"8effc54f-fa47-4e08-88ee-c49ea54855db","Type":"ContainerStarted","Data":"28ab61a0b148c8134ebbb10cd61568b2a37a76a87d747a8565b83f6bf8af1274"} Jul 22 00:42:05.016824 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:05.016745 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" podStartSLOduration=1.8157809249999999 podStartE2EDuration="2.016728528s" totalImagesPullingTime="200.947603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:42:05.015311586 +0000 UTC m=+933.341383790" watchObservedRunningTime="2026-07-22 00:42:05.016728528 +0000 UTC m=+933.342800730" Jul 22 00:42:13.818485 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:13.818433 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:13.819010 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:13.818519 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:13.831382 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:13.831358 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:14.055892 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:14.055856 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4pcs" Jul 22 00:42:22.632859 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:42:22.632823 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:43:09.235007 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:43:09.234971 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:43:37.197047 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:43:37.197005 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:43:37.897456 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:43:37.897415 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:44:05.757651 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:05.757616 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-586fc58b47-894bc"] Jul 22 00:44:05.761118 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:05.761100 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586fc58b47-894bc" Jul 22 00:44:05.775383 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:05.775355 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-586fc58b47-894bc"] Jul 22 00:44:05.884771 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:05.884724 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5193d573-58ee-4ea7-b823-c8ffc05cd95e-tls-cert\") pod \"authorino-586fc58b47-894bc\" (UID: \"5193d573-58ee-4ea7-b823-c8ffc05cd95e\") " pod="kuadrant-system/authorino-586fc58b47-894bc" Jul 22 00:44:05.884771 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:05.884769 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8t9d\" (UniqueName: \"kubernetes.io/projected/5193d573-58ee-4ea7-b823-c8ffc05cd95e-kube-api-access-t8t9d\") pod \"authorino-586fc58b47-894bc\" (UID: \"5193d573-58ee-4ea7-b823-c8ffc05cd95e\") " pod="kuadrant-system/authorino-586fc58b47-894bc" Jul 22 00:44:05.985603 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:05.985554 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5193d573-58ee-4ea7-b823-c8ffc05cd95e-tls-cert\") pod \"authorino-586fc58b47-894bc\" (UID: \"5193d573-58ee-4ea7-b823-c8ffc05cd95e\") " pod="kuadrant-system/authorino-586fc58b47-894bc" Jul 22 00:44:05.985603 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:05.985596 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8t9d\" (UniqueName: \"kubernetes.io/projected/5193d573-58ee-4ea7-b823-c8ffc05cd95e-kube-api-access-t8t9d\") pod \"authorino-586fc58b47-894bc\" (UID: \"5193d573-58ee-4ea7-b823-c8ffc05cd95e\") " pod="kuadrant-system/authorino-586fc58b47-894bc" Jul 22 00:44:05.988055 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:05.988029 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5193d573-58ee-4ea7-b823-c8ffc05cd95e-tls-cert\") pod \"authorino-586fc58b47-894bc\" (UID: \"5193d573-58ee-4ea7-b823-c8ffc05cd95e\") " pod="kuadrant-system/authorino-586fc58b47-894bc" Jul 22 00:44:05.992748 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:05.992724 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8t9d\" (UniqueName: \"kubernetes.io/projected/5193d573-58ee-4ea7-b823-c8ffc05cd95e-kube-api-access-t8t9d\") pod \"authorino-586fc58b47-894bc\" (UID: \"5193d573-58ee-4ea7-b823-c8ffc05cd95e\") " pod="kuadrant-system/authorino-586fc58b47-894bc" Jul 22 00:44:06.069975 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:06.069933 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586fc58b47-894bc" Jul 22 00:44:06.092438 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:06.092403 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:44:06.191753 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:06.191716 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:44:06.207554 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:06.207521 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-586fc58b47-894bc"] Jul 22 00:44:06.211276 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:44:06.211240 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5193d573_58ee_4ea7_b823_c8ffc05cd95e.slice/crio-61610427d254eef0ce63b899168119a5c976cf2a977d46033151a7adae67eaf6 WatchSource:0}: Error finding container 61610427d254eef0ce63b899168119a5c976cf2a977d46033151a7adae67eaf6: Status 404 returned error can't find the container with id 61610427d254eef0ce63b899168119a5c976cf2a977d46033151a7adae67eaf6 Jul 22 00:44:06.587724 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:06.587689 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586fc58b47-894bc" event={"ID":"5193d573-58ee-4ea7-b823-c8ffc05cd95e","Type":"ContainerStarted","Data":"61610427d254eef0ce63b899168119a5c976cf2a977d46033151a7adae67eaf6"} Jul 22 00:44:07.303269 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:07.303225 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:44:07.406361 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:07.406262 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:44:07.596000 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:07.595960 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586fc58b47-894bc" event={"ID":"5193d573-58ee-4ea7-b823-c8ffc05cd95e","Type":"ContainerStarted","Data":"8d2e1b96f79f567a83042d30b492d0a8c06d42754c24a584f811e436d10a4205"} Jul 22 00:44:07.613718 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:07.613667 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-586fc58b47-894bc" podStartSLOduration=1.697810009 podStartE2EDuration="2.613646997s" totalImagesPullingTime="915.836988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:44:07.61357915 +0000 UTC m=+1055.939651377" watchObservedRunningTime="2026-07-22 00:44:07.613646997 +0000 UTC m=+1055.939719198" Jul 22 00:44:07.640732 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:07.640686 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d7f8b89d-66pwl"] Jul 22 00:44:07.640976 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:07.640941 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d7f8b89d-66pwl" podUID="0755509d-5f71-4838-b096-e66d16d861a1" containerName="authorino" containerID="cri-o://85ac59df5868b312f87fb5d85bf080d1344b492f325d90a417155fea16c89654" gracePeriod=30 Jul 22 00:44:07.907072 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:07.907045 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d7f8b89d-66pwl" Jul 22 00:44:08.004279 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:08.004176 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2grhs\" (UniqueName: \"kubernetes.io/projected/0755509d-5f71-4838-b096-e66d16d861a1-kube-api-access-2grhs\") pod \"0755509d-5f71-4838-b096-e66d16d861a1\" (UID: \"0755509d-5f71-4838-b096-e66d16d861a1\") " Jul 22 00:44:08.004458 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:08.004341 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0755509d-5f71-4838-b096-e66d16d861a1-tls-cert\") pod \"0755509d-5f71-4838-b096-e66d16d861a1\" (UID: \"0755509d-5f71-4838-b096-e66d16d861a1\") " Jul 22 00:44:08.006455 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:08.006422 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0755509d-5f71-4838-b096-e66d16d861a1-kube-api-access-2grhs" (OuterVolumeSpecName: "kube-api-access-2grhs") pod "0755509d-5f71-4838-b096-e66d16d861a1" (UID: "0755509d-5f71-4838-b096-e66d16d861a1"). InnerVolumeSpecName "kube-api-access-2grhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:44:08.015823 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:08.015747 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0755509d-5f71-4838-b096-e66d16d861a1-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0755509d-5f71-4838-b096-e66d16d861a1" (UID: "0755509d-5f71-4838-b096-e66d16d861a1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:44:08.105452 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:08.105409 2567 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0755509d-5f71-4838-b096-e66d16d861a1-tls-cert\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:44:08.105452 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:08.105445 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2grhs\" (UniqueName: \"kubernetes.io/projected/0755509d-5f71-4838-b096-e66d16d861a1-kube-api-access-2grhs\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:44:08.602636 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:08.602603 2567 generic.go:356] "Generic (PLEG): container finished" podID="0755509d-5f71-4838-b096-e66d16d861a1" containerID="85ac59df5868b312f87fb5d85bf080d1344b492f325d90a417155fea16c89654" exitCode=0 Jul 22 00:44:08.603079 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:08.602650 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d7f8b89d-66pwl" Jul 22 00:44:08.603079 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:08.602695 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d7f8b89d-66pwl" event={"ID":"0755509d-5f71-4838-b096-e66d16d861a1","Type":"ContainerDied","Data":"85ac59df5868b312f87fb5d85bf080d1344b492f325d90a417155fea16c89654"} Jul 22 00:44:08.603079 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:08.602745 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d7f8b89d-66pwl" event={"ID":"0755509d-5f71-4838-b096-e66d16d861a1","Type":"ContainerDied","Data":"13356dbb29536f84596fbbe8fe9c61aa90afd14d04c54895dd504df7d5129c0e"} Jul 22 00:44:08.603079 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:08.602772 2567 scope.go:117] "RemoveContainer" containerID="85ac59df5868b312f87fb5d85bf080d1344b492f325d90a417155fea16c89654" Jul 22 00:44:08.615368 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:08.613533 2567 scope.go:117] "RemoveContainer" containerID="85ac59df5868b312f87fb5d85bf080d1344b492f325d90a417155fea16c89654" Jul 22 00:44:08.615536 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:44:08.615458 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ac59df5868b312f87fb5d85bf080d1344b492f325d90a417155fea16c89654\": container with ID starting with 85ac59df5868b312f87fb5d85bf080d1344b492f325d90a417155fea16c89654 not found: ID does not exist" containerID="85ac59df5868b312f87fb5d85bf080d1344b492f325d90a417155fea16c89654" Jul 22 00:44:08.615536 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:08.615498 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ac59df5868b312f87fb5d85bf080d1344b492f325d90a417155fea16c89654"} err="failed to get container status \"85ac59df5868b312f87fb5d85bf080d1344b492f325d90a417155fea16c89654\": rpc error: code = NotFound desc = could not find container \"85ac59df5868b312f87fb5d85bf080d1344b492f325d90a417155fea16c89654\": container with ID starting with 85ac59df5868b312f87fb5d85bf080d1344b492f325d90a417155fea16c89654 not found: ID does not exist" Jul 22 00:44:08.618998 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:08.618958 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d7f8b89d-66pwl"] Jul 22 00:44:08.622462 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:08.622433 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d7f8b89d-66pwl"] Jul 22 00:44:10.119392 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:10.119358 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0755509d-5f71-4838-b096-e66d16d861a1" path="/var/lib/kubelet/pods/0755509d-5f71-4838-b096-e66d16d861a1/volumes" Jul 22 00:44:34.647229 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:34.647189 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:44:47.738008 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:44:47.737963 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:45:43.061108 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:45:43.061069 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:46:12.653292 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:46:12.653260 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:46:13.904185 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:46:13.904146 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:46:40.122427 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:46:40.122397 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:46:40.124890 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:46:40.124450 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:46:40.247683 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:46:40.247639 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:46:40.348142 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:46:40.348085 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:46:50.715085 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:46:50.715040 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:46:50.812929 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:46:50.812890 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:46:51.048839 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:46:51.048802 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:46:56.761533 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:46:56.761494 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:46:58.142893 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:46:58.142858 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:47:02.761271 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:47:02.761229 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:47:02.860469 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:47:02.860434 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:47:45.978395 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:47:45.978355 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:50:46.367296 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:50:46.367247 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:50:56.189232 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:50:56.189193 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:51:40.152698 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:51:40.152575 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:51:40.157477 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:51:40.157450 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:52:51.347739 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:52:51.347695 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:52:53.861156 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:52:53.861123 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:54:39.998076 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:39.998034 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:54:42.506888 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.506850 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295"] Jul 22 00:54:42.507476 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.507454 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0755509d-5f71-4838-b096-e66d16d861a1" containerName="authorino" Jul 22 00:54:42.507562 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.507479 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0755509d-5f71-4838-b096-e66d16d861a1" containerName="authorino" Jul 22 00:54:42.507630 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.507576 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0755509d-5f71-4838-b096-e66d16d861a1" containerName="authorino" Jul 22 00:54:42.511451 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.511428 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295"] Jul 22 00:54:42.511583 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.511509 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.514135 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.514109 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-7e056d38-kserve-self-signed-certs\"" Jul 22 00:54:42.532643 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.532590 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:54:42.614979 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.614931 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-model-cache\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.614979 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.614983 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5l9j\" (UniqueName: \"kubernetes.io/projected/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-kube-api-access-z5l9j\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.615223 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.615080 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-tmp-dir\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.615223 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.615135 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-dshm\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.615223 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.615155 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-home\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.615223 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.615216 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-tls-certs\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.716035 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.715991 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5l9j\" (UniqueName: \"kubernetes.io/projected/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-kube-api-access-z5l9j\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.716240 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.716057 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-tmp-dir\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.716240 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.716105 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-dshm\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.716240 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.716216 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-home\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.716421 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.716304 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-tls-certs\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.716421 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.716365 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-model-cache\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.716668 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.716637 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-home\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.716825 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.716639 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-tmp-dir\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.716825 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.716651 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-model-cache\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.718551 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.718521 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-dshm\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.718721 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.718702 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-tls-certs\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.724662 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.724636 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5l9j\" (UniqueName: \"kubernetes.io/projected/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-kube-api-access-z5l9j\") pod \"e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.821965 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.821924 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:42.974221 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:42.974188 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295"] Jul 22 00:54:42.976413 ip-10-0-132-178 kubenswrapper[2567]: W0722 00:54:42.976376 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9b1b657_d7c8_450a_b979_551f4b4ae8a5.slice/crio-496c7a851dad02ad86a9885d180c4ddfeb256a51046fb8df1f74cd54e8b88bb9 WatchSource:0}: Error finding container 496c7a851dad02ad86a9885d180c4ddfeb256a51046fb8df1f74cd54e8b88bb9: Status 404 returned error can't find the container with id 496c7a851dad02ad86a9885d180c4ddfeb256a51046fb8df1f74cd54e8b88bb9 Jul 22 00:54:43.782359 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:43.782318 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" event={"ID":"b9b1b657-d7c8-450a-b979-551f4b4ae8a5","Type":"ContainerStarted","Data":"f73966281d927df45e5aca6d0751a097ba305767abf16643221ba90f3430c9d7"} Jul 22 00:54:43.782748 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:43.782368 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" event={"ID":"b9b1b657-d7c8-450a-b979-551f4b4ae8a5","Type":"ContainerStarted","Data":"496c7a851dad02ad86a9885d180c4ddfeb256a51046fb8df1f74cd54e8b88bb9"} Jul 22 00:54:43.803422 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:43.803363 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" podStartSLOduration=1.8033468639999999 podStartE2EDuration="1.803346864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:54:43.80236563 +0000 UTC m=+1692.128437847" watchObservedRunningTime="2026-07-22 00:54:43.803346864 +0000 UTC m=+1692.129419065" Jul 22 00:54:44.753536 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:44.753494 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:54:52.822106 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:52.822063 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:52.822570 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:52.822135 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:52.835334 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:52.835304 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:54:53.845470 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:54:53.845439 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:55:06.006281 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:06.006174 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:55:07.079929 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:07.079894 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295"] Jul 22 00:55:07.100613 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:07.100578 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:55:22.100999 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.100957 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" podUID="b9b1b657-d7c8-450a-b979-551f4b4ae8a5" containerName="main" containerID="cri-o://f73966281d927df45e5aca6d0751a097ba305767abf16643221ba90f3430c9d7" gracePeriod=45 Jul 22 00:55:22.354704 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.354626 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:55:22.401868 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.401837 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-home\") pod \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " Jul 22 00:55:22.402052 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.401901 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-dshm\") pod \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " Jul 22 00:55:22.402052 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.401928 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5l9j\" (UniqueName: \"kubernetes.io/projected/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-kube-api-access-z5l9j\") pod \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " Jul 22 00:55:22.402052 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.401946 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-tls-certs\") pod \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " Jul 22 00:55:22.402052 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.402017 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-tmp-dir\") pod \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " Jul 22 00:55:22.402278 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.402060 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-model-cache\") pod \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\" (UID: \"b9b1b657-d7c8-450a-b979-551f4b4ae8a5\") " Jul 22 00:55:22.402278 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.402150 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-home" (OuterVolumeSpecName: "home") pod "b9b1b657-d7c8-450a-b979-551f4b4ae8a5" (UID: "b9b1b657-d7c8-450a-b979-551f4b4ae8a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:55:22.402381 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.402332 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9b1b657-d7c8-450a-b979-551f4b4ae8a5" (UID: "b9b1b657-d7c8-450a-b979-551f4b4ae8a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:55:22.402381 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.402356 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9b1b657-d7c8-450a-b979-551f4b4ae8a5" (UID: "b9b1b657-d7c8-450a-b979-551f4b4ae8a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:55:22.402381 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.402374 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-home\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:55:22.404319 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.404287 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-dshm" (OuterVolumeSpecName: "dshm") pod "b9b1b657-d7c8-450a-b979-551f4b4ae8a5" (UID: "b9b1b657-d7c8-450a-b979-551f4b4ae8a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:55:22.404319 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.404295 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9b1b657-d7c8-450a-b979-551f4b4ae8a5" (UID: "b9b1b657-d7c8-450a-b979-551f4b4ae8a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:55:22.404462 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.404397 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-kube-api-access-z5l9j" (OuterVolumeSpecName: "kube-api-access-z5l9j") pod "b9b1b657-d7c8-450a-b979-551f4b4ae8a5" (UID: "b9b1b657-d7c8-450a-b979-551f4b4ae8a5"). InnerVolumeSpecName "kube-api-access-z5l9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:55:22.502937 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.502905 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5l9j\" (UniqueName: \"kubernetes.io/projected/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-kube-api-access-z5l9j\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:55:22.502937 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.502932 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-tls-certs\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:55:22.503228 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.502948 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-tmp-dir\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:55:22.503228 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.502961 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-model-cache\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:55:22.503228 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.502969 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b1b657-d7c8-450a-b979-551f4b4ae8a5-dshm\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 00:55:22.978088 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.978050 2567 generic.go:356] "Generic (PLEG): container finished" podID="b9b1b657-d7c8-450a-b979-551f4b4ae8a5" containerID="f73966281d927df45e5aca6d0751a097ba305767abf16643221ba90f3430c9d7" exitCode=0 Jul 22 00:55:22.978272 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.978097 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" Jul 22 00:55:22.978272 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.978126 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" event={"ID":"b9b1b657-d7c8-450a-b979-551f4b4ae8a5","Type":"ContainerDied","Data":"f73966281d927df45e5aca6d0751a097ba305767abf16643221ba90f3430c9d7"} Jul 22 00:55:22.978272 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.978173 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295" event={"ID":"b9b1b657-d7c8-450a-b979-551f4b4ae8a5","Type":"ContainerDied","Data":"496c7a851dad02ad86a9885d180c4ddfeb256a51046fb8df1f74cd54e8b88bb9"} Jul 22 00:55:22.978272 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.978192 2567 scope.go:117] "RemoveContainer" containerID="f73966281d927df45e5aca6d0751a097ba305767abf16643221ba90f3430c9d7" Jul 22 00:55:22.987673 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.987652 2567 scope.go:117] "RemoveContainer" containerID="f73966281d927df45e5aca6d0751a097ba305767abf16643221ba90f3430c9d7" Jul 22 00:55:22.988033 ip-10-0-132-178 kubenswrapper[2567]: E0722 00:55:22.988008 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73966281d927df45e5aca6d0751a097ba305767abf16643221ba90f3430c9d7\": container with ID starting with f73966281d927df45e5aca6d0751a097ba305767abf16643221ba90f3430c9d7 not found: ID does not exist" containerID="f73966281d927df45e5aca6d0751a097ba305767abf16643221ba90f3430c9d7" Jul 22 00:55:22.988139 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.988041 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73966281d927df45e5aca6d0751a097ba305767abf16643221ba90f3430c9d7"} err="failed to get container status \"f73966281d927df45e5aca6d0751a097ba305767abf16643221ba90f3430c9d7\": rpc error: code = NotFound desc = could not find container \"f73966281d927df45e5aca6d0751a097ba305767abf16643221ba90f3430c9d7\": container with ID starting with f73966281d927df45e5aca6d0751a097ba305767abf16643221ba90f3430c9d7 not found: ID does not exist" Jul 22 00:55:22.999968 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:22.999926 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295"] Jul 22 00:55:23.002046 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:23.002019 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-7e056d38-kserve-8855f65f9-bg295"] Jul 22 00:55:24.119272 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:55:24.119236 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9b1b657-d7c8-450a-b979-551f4b4ae8a5" path="/var/lib/kubelet/pods/b9b1b657-d7c8-450a-b979-551f4b4ae8a5/volumes" Jul 22 00:56:40.182534 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:56:40.182417 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:56:40.189391 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:56:40.189370 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 00:58:13.649790 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:13.649678 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:58:35.173433 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:35.173394 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:58:38.267268 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:38.267215 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:58:38.364385 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:38.364337 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:58:38.729144 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:38.729044 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:58:44.456341 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:44.456303 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:58:44.556786 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:44.556751 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:58:44.699034 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:44.698997 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:58:45.012028 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:45.011994 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:58:45.534175 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:45.534129 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:58:47.228601 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:47.228562 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:58:47.379343 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:47.379299 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:58:47.495541 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:47.495448 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:58:47.594337 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:47.594299 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:58:47.895658 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:47.895608 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:58:47.995912 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:58:47.995876 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:59:23.170843 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:59:23.170801 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:59:27.853269 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:59:27.853185 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:59:28.005705 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:59:28.005671 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:59:28.241891 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:59:28.241801 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:59:28.492311 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:59:28.492203 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:59:28.591231 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:59:28.591176 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:59:28.695372 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:59:28.695336 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:59:29.006317 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:59:29.006275 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:59:29.363742 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:59:29.363706 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:59:31.556699 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:59:31.556664 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:59:31.691321 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:59:31.691274 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:59:31.813253 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:59:31.813161 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:59:32.153289 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:59:32.153189 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 00:59:32.252524 ip-10-0-132-178 kubenswrapper[2567]: I0722 00:59:32.252478 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:07.256483 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:07.256442 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:11.631281 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:11.631246 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:11.788576 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:11.788543 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:12.025238 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:12.025183 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:12.285511 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:12.285424 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:12.608366 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:12.608274 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:12.955466 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:12.955373 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:14.689476 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:14.689434 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:14.777212 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:14.777179 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:14.839437 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:14.839405 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:15.204128 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:15.204092 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:15.515899 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:15.515865 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:15.755064 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:15.755026 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:36.415034 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:36.414994 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:36.472502 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:36.472471 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:50.914971 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:50.914922 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:53.899687 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:53.899643 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:53.953865 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:53.953829 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:55.579410 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:55.579368 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:56.821341 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:56.821305 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:56.984495 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:56.984459 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:57.226884 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:57.226810 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:59.579303 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:59.579252 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:59.634581 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:59.634548 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:00:59.970369 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:00:59.970274 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:00.069761 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:00.069733 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:00.221885 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:00.221807 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:00.811968 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:00.811934 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:00.916407 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:00.916370 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:01.015794 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:01.015740 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:35.492447 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:35.492412 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:37.970022 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:37.969977 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:38.088048 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:38.088021 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:39.917156 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:39.917093 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:40.083116 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:40.083082 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:40.212947 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:40.212799 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 01:01:40.220436 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:40.220417 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 01:01:40.299449 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:40.299423 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:40.821583 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:40.821553 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:40.915027 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:40.914994 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:41.016445 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:41.016405 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:42.475728 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:42.475658 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:48.924506 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:48.924463 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:49.045314 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:49.045269 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:49.793690 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:49.793652 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:01:49.921522 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:01:49.921489 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:02:04.118103 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:02:04.118062 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:02:15.670514 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:02:15.670482 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:02:17.795952 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:02:17.795905 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:02:23.654594 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:02:23.654555 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:02:24.210548 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:02:24.210508 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:02:24.375862 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:02:24.375826 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:02:39.885629 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:02:39.885583 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:02:46.338172 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:02:46.338136 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:02:46.490770 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:02:46.490718 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:02:46.681656 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:02:46.681554 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:02:57.083698 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:02:57.083659 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:02:57.910614 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:02:57.910581 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:05.921629 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:05.921585 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:06.038487 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:06.038452 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:06.146674 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:06.146635 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:06.255217 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:06.255117 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:07.088992 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:07.088955 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:07.209680 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:07.209634 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:09.026901 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:09.026862 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:15.274194 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:15.274153 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:15.369774 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:15.369733 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:15.923679 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:15.923642 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:22.523936 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:22.523894 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:27.573098 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:27.573055 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:30.660610 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:30.660569 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:31.486257 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:31.486216 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:31.638089 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:31.638052 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:31.860342 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:31.860306 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:37.766884 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:37.766844 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:37.866255 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:37.866219 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:48.790057 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:48.790016 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:53.201987 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:53.201948 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:53.611838 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:53.611796 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:59.545397 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:59.545302 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:59.646148 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:59.646111 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:03:59.826256 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:03:59.826171 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:00.131066 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:00.130980 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:00.225881 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:00.225847 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:00.399064 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:00.398967 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:06.200418 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:06.200377 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:06.329468 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:06.329422 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:06.458758 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:06.458651 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:06.575714 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:06.575672 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:12.506615 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:12.506566 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:12.610105 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:12.610066 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:19.328387 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:19.328344 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:19.681395 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:19.681302 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:20.935471 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:20.935435 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:22.387966 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:22.387927 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:22.489336 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:22.489304 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:22.558833 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:22.558794 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:23.060442 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:23.060401 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:23.162000 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:23.161970 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:23.294569 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:23.294531 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:26.919827 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:26.919795 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:32.533115 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.533070 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw"] Jul 22 01:04:32.533651 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.533629 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b1b657-d7c8-450a-b979-551f4b4ae8a5" containerName="main" Jul 22 01:04:32.533735 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.533654 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b1b657-d7c8-450a-b979-551f4b4ae8a5" containerName="main" Jul 22 01:04:32.533823 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.533765 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b1b657-d7c8-450a-b979-551f4b4ae8a5" containerName="main" Jul 22 01:04:32.536958 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.536930 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.541011 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.540989 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c9532535\"/\"default-dockercfg-gzp5p\"" Jul 22 01:04:32.541011 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.541002 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c9532535\"/\"kube-root-ca.crt\"" Jul 22 01:04:32.541221 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.541002 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c9532535\"/\"openshift-service-ca.crt\"" Jul 22 01:04:32.541221 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.541015 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c9532535\"/\"e2e-lifecycle-model-c9532535-kserve-self-signed-certs\"" Jul 22 01:04:32.547845 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.547819 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw"] Jul 22 01:04:32.642323 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.642267 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:04:32.669616 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.669580 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-home\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.669616 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.669616 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9jn7\" (UniqueName: \"kubernetes.io/projected/7686dcd6-6d63-4b17-927f-e01fc823e9c8-kube-api-access-p9jn7\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.669855 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.669730 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-dshm\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.669855 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.669758 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-tmp-dir\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.669855 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.669799 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7686dcd6-6d63-4b17-927f-e01fc823e9c8-tls-certs\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.669855 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.669825 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-model-cache\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.770461 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.770430 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-dshm\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.770643 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.770465 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-tmp-dir\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.770643 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.770595 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7686dcd6-6d63-4b17-927f-e01fc823e9c8-tls-certs\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.770792 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.770640 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-model-cache\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.770859 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.770790 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-home\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.770859 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.770823 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9jn7\" (UniqueName: \"kubernetes.io/projected/7686dcd6-6d63-4b17-927f-e01fc823e9c8-kube-api-access-p9jn7\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.770974 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.770957 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-tmp-dir\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.771120 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.771097 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-model-cache\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.771190 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.771162 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-home\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.772924 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.772902 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-dshm\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.773234 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.773217 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7686dcd6-6d63-4b17-927f-e01fc823e9c8-tls-certs\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.779820 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.779770 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9jn7\" (UniqueName: \"kubernetes.io/projected/7686dcd6-6d63-4b17-927f-e01fc823e9c8-kube-api-access-p9jn7\") pod \"e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.847104 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.847007 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:32.992625 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:32.986249 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw"] Jul 22 01:04:32.992625 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:04:32.990150 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7686dcd6_6d63_4b17_927f_e01fc823e9c8.slice/crio-bdb56b5b89bc91bb0f9016580e7bc4075c0663ea2b6928ffd2e637796e78aa5b WatchSource:0}: Error finding container bdb56b5b89bc91bb0f9016580e7bc4075c0663ea2b6928ffd2e637796e78aa5b: Status 404 returned error can't find the container with id bdb56b5b89bc91bb0f9016580e7bc4075c0663ea2b6928ffd2e637796e78aa5b Jul 22 01:04:33.727168 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:33.727133 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" event={"ID":"7686dcd6-6d63-4b17-927f-e01fc823e9c8","Type":"ContainerStarted","Data":"470559ff8196b78d863405e898c9ae9a3dc004779a0f294e0c64c20f6f921fd7"} Jul 22 01:04:33.727168 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:33.727175 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" event={"ID":"7686dcd6-6d63-4b17-927f-e01fc823e9c8","Type":"ContainerStarted","Data":"bdb56b5b89bc91bb0f9016580e7bc4075c0663ea2b6928ffd2e637796e78aa5b"} Jul 22 01:04:33.745796 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:33.745729 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" podStartSLOduration=1.745713396 podStartE2EDuration="1.745713396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:04:33.74432719 +0000 UTC m=+2282.070399415" watchObservedRunningTime="2026-07-22 01:04:33.745713396 +0000 UTC m=+2282.071785597" Jul 22 01:04:42.847619 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:42.847572 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:42.848185 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:42.847657 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:42.860299 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:42.860274 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:04:43.793226 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:04:43.793191 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:05:10.528800 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:10.528740 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:05:15.189597 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:15.189562 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:05:17.457728 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:17.457684 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:05:22.410236 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:22.410196 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw"] Jul 22 01:05:22.507303 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:22.507264 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:05:37.431799 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.431730 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" podUID="7686dcd6-6d63-4b17-927f-e01fc823e9c8" containerName="main" containerID="cri-o://470559ff8196b78d863405e898c9ae9a3dc004779a0f294e0c64c20f6f921fd7" gracePeriod=45 Jul 22 01:05:37.691157 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.691091 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:05:37.786717 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.786661 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-model-cache\") pod \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " Jul 22 01:05:37.786942 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.786773 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-tmp-dir\") pod \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " Jul 22 01:05:37.786942 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.786820 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7686dcd6-6d63-4b17-927f-e01fc823e9c8-tls-certs\") pod \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " Jul 22 01:05:37.786942 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.786859 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9jn7\" (UniqueName: \"kubernetes.io/projected/7686dcd6-6d63-4b17-927f-e01fc823e9c8-kube-api-access-p9jn7\") pod \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " Jul 22 01:05:37.786942 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.786910 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-dshm\") pod \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " Jul 22 01:05:37.787140 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.786945 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-home\") pod \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\" (UID: \"7686dcd6-6d63-4b17-927f-e01fc823e9c8\") " Jul 22 01:05:37.787140 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.786979 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7686dcd6-6d63-4b17-927f-e01fc823e9c8" (UID: "7686dcd6-6d63-4b17-927f-e01fc823e9c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:05:37.787140 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.787064 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7686dcd6-6d63-4b17-927f-e01fc823e9c8" (UID: "7686dcd6-6d63-4b17-927f-e01fc823e9c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:05:37.787300 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.787266 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-home" (OuterVolumeSpecName: "home") pod "7686dcd6-6d63-4b17-927f-e01fc823e9c8" (UID: "7686dcd6-6d63-4b17-927f-e01fc823e9c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:05:37.787419 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.787304 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-tmp-dir\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:05:37.787419 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.787322 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-model-cache\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:05:37.789101 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.789078 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-dshm" (OuterVolumeSpecName: "dshm") pod "7686dcd6-6d63-4b17-927f-e01fc823e9c8" (UID: "7686dcd6-6d63-4b17-927f-e01fc823e9c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:05:37.789184 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.789155 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7686dcd6-6d63-4b17-927f-e01fc823e9c8-kube-api-access-p9jn7" (OuterVolumeSpecName: "kube-api-access-p9jn7") pod "7686dcd6-6d63-4b17-927f-e01fc823e9c8" (UID: "7686dcd6-6d63-4b17-927f-e01fc823e9c8"). InnerVolumeSpecName "kube-api-access-p9jn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:05:37.789227 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.789213 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7686dcd6-6d63-4b17-927f-e01fc823e9c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7686dcd6-6d63-4b17-927f-e01fc823e9c8" (UID: "7686dcd6-6d63-4b17-927f-e01fc823e9c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:05:37.888358 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.888318 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7686dcd6-6d63-4b17-927f-e01fc823e9c8-tls-certs\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:05:37.888358 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.888352 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9jn7\" (UniqueName: \"kubernetes.io/projected/7686dcd6-6d63-4b17-927f-e01fc823e9c8-kube-api-access-p9jn7\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:05:37.888358 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.888362 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-dshm\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:05:37.888602 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:37.888371 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7686dcd6-6d63-4b17-927f-e01fc823e9c8-home\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:05:38.052861 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:38.052816 2567 generic.go:356] "Generic (PLEG): container finished" podID="7686dcd6-6d63-4b17-927f-e01fc823e9c8" containerID="470559ff8196b78d863405e898c9ae9a3dc004779a0f294e0c64c20f6f921fd7" exitCode=0 Jul 22 01:05:38.052861 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:38.052867 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" Jul 22 01:05:38.053092 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:38.052901 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" event={"ID":"7686dcd6-6d63-4b17-927f-e01fc823e9c8","Type":"ContainerDied","Data":"470559ff8196b78d863405e898c9ae9a3dc004779a0f294e0c64c20f6f921fd7"} Jul 22 01:05:38.053092 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:38.052949 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw" event={"ID":"7686dcd6-6d63-4b17-927f-e01fc823e9c8","Type":"ContainerDied","Data":"bdb56b5b89bc91bb0f9016580e7bc4075c0663ea2b6928ffd2e637796e78aa5b"} Jul 22 01:05:38.053092 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:38.052968 2567 scope.go:117] "RemoveContainer" containerID="470559ff8196b78d863405e898c9ae9a3dc004779a0f294e0c64c20f6f921fd7" Jul 22 01:05:38.062017 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:38.061995 2567 scope.go:117] "RemoveContainer" containerID="470559ff8196b78d863405e898c9ae9a3dc004779a0f294e0c64c20f6f921fd7" Jul 22 01:05:38.062307 ip-10-0-132-178 kubenswrapper[2567]: E0722 01:05:38.062287 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470559ff8196b78d863405e898c9ae9a3dc004779a0f294e0c64c20f6f921fd7\": container with ID starting with 470559ff8196b78d863405e898c9ae9a3dc004779a0f294e0c64c20f6f921fd7 not found: ID does not exist" containerID="470559ff8196b78d863405e898c9ae9a3dc004779a0f294e0c64c20f6f921fd7" Jul 22 01:05:38.062370 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:38.062320 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470559ff8196b78d863405e898c9ae9a3dc004779a0f294e0c64c20f6f921fd7"} err="failed to get container status \"470559ff8196b78d863405e898c9ae9a3dc004779a0f294e0c64c20f6f921fd7\": rpc error: code = NotFound desc = could not find container \"470559ff8196b78d863405e898c9ae9a3dc004779a0f294e0c64c20f6f921fd7\": container with ID starting with 470559ff8196b78d863405e898c9ae9a3dc004779a0f294e0c64c20f6f921fd7 not found: ID does not exist" Jul 22 01:05:38.075081 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:38.075044 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw"] Jul 22 01:05:38.077490 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:38.077466 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c9532535/e2e-lifecycle-model-c9532535-kserve-7d58c6cc9f-b5cqw"] Jul 22 01:05:38.119377 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:38.119342 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7686dcd6-6d63-4b17-927f-e01fc823e9c8" path="/var/lib/kubelet/pods/7686dcd6-6d63-4b17-927f-e01fc823e9c8/volumes" Jul 22 01:05:58.662295 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:58.662258 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:05:59.485984 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:05:59.485937 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:05.125914 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:05.125877 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:05.226607 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:05.226572 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:05.325454 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:05.325416 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:05.571725 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:05.571697 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:06.631118 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:06.631080 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:06.734677 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:06.734635 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:06.834737 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:06.834706 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:07.548514 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:07.548481 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:07.645883 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:07.645845 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:07.745223 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:07.745197 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:09.073341 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:09.073302 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:14.683049 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:14.683010 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:14.764564 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:14.764513 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:15.334435 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:15.334388 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:21.233440 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:21.233388 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:21.333206 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:21.333173 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:21.475188 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:21.475151 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:29.439632 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:29.439594 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:31.943715 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:31.943674 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:32.337210 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:32.337181 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:38.752117 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:38.752078 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:38.850818 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:38.850757 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:40.243406 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:40.243292 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 01:06:40.251446 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:40.251428 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 01:06:53.206855 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:53.206804 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:59.676135 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:59.676053 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:06:59.776920 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:06:59.776888 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:00.015472 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:00.015440 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:00.328897 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:00.328814 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:00.436581 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:00.436534 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:03.274225 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:03.274194 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:03.573485 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:03.573379 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:03.677268 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:03.677231 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:04.044051 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:04.044014 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:04.143173 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:04.143133 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:09.605410 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:09.605373 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:09.721172 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:09.721136 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:11.205956 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:11.205918 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:11.289791 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:11.289748 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:11.363983 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:11.363944 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:11.465010 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:11.464922 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:11.843641 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:11.843606 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:11.973911 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:11.973880 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:15.746755 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:15.746690 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:24.922301 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:24.922264 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:24.928922 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:24.928888 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h"] Jul 22 01:07:24.929425 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:24.929401 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7686dcd6-6d63-4b17-927f-e01fc823e9c8" containerName="main" Jul 22 01:07:24.929425 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:24.929425 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7686dcd6-6d63-4b17-927f-e01fc823e9c8" containerName="main" Jul 22 01:07:24.929582 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:24.929532 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7686dcd6-6d63-4b17-927f-e01fc823e9c8" containerName="main" Jul 22 01:07:24.932756 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:24.932736 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:24.935561 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:24.935541 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-805a5b\"/\"kube-root-ca.crt\"" Jul 22 01:07:24.936901 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:24.936878 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-805a5b\"/\"default-dockercfg-gqfx7\"" Jul 22 01:07:24.937008 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:24.936907 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-805a5b\"/\"openshift-service-ca.crt\"" Jul 22 01:07:24.937008 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:24.936881 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-805a5b\"/\"test-model-805a5b-kserve-self-signed-certs\"" Jul 22 01:07:24.943057 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:24.943034 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h"] Jul 22 01:07:25.020520 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.020484 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:25.085768 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.085727 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-dshm\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.086000 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.085814 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-tmp-dir\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.086000 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.085862 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bnzs\" (UniqueName: \"kubernetes.io/projected/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-kube-api-access-6bnzs\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.086000 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.085944 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-tls-certs\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.086163 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.086019 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-model-cache\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.086163 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.086063 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-home\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.187269 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.187170 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bnzs\" (UniqueName: \"kubernetes.io/projected/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-kube-api-access-6bnzs\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.187269 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.187225 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-tls-certs\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.187269 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.187270 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-model-cache\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.187586 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.187298 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-home\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.187586 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.187323 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-dshm\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.187586 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.187445 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-tmp-dir\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.187844 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.187818 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-model-cache\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.187898 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.187842 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-home\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.187949 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.187890 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-tmp-dir\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.189674 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.189654 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-dshm\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.189859 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.189842 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-tls-certs\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.195174 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.195149 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bnzs\" (UniqueName: \"kubernetes.io/projected/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-kube-api-access-6bnzs\") pod \"test-model-805a5b-kserve-647f747547-8p49h\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.242366 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.242332 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:25.375855 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.375821 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h"] Jul 22 01:07:25.378608 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:07:25.378579 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c0a77c2_f94a_438b_9b26_52b9678b1fa8.slice/crio-c404adf4fa964e270c7112aa98c49497247d2628844c366fe26cbb5f9fcf4f39 WatchSource:0}: Error finding container c404adf4fa964e270c7112aa98c49497247d2628844c366fe26cbb5f9fcf4f39: Status 404 returned error can't find the container with id c404adf4fa964e270c7112aa98c49497247d2628844c366fe26cbb5f9fcf4f39 Jul 22 01:07:25.575730 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.575696 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" event={"ID":"8c0a77c2-f94a-438b-9b26-52b9678b1fa8","Type":"ContainerStarted","Data":"90ebf297467e4244108a96251964d6774ad5314e774d1731de038cab8b88d750"} Jul 22 01:07:25.575730 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.575739 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" event={"ID":"8c0a77c2-f94a-438b-9b26-52b9678b1fa8","Type":"ContainerStarted","Data":"c404adf4fa964e270c7112aa98c49497247d2628844c366fe26cbb5f9fcf4f39"} Jul 22 01:07:25.593873 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:25.593820 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" podStartSLOduration=1.593804146 podStartE2EDuration="1.593804146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:07:25.592518188 +0000 UTC m=+2453.918590391" watchObservedRunningTime="2026-07-22 01:07:25.593804146 +0000 UTC m=+2453.919876346" Jul 22 01:07:32.680728 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:32.680426 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:33.174573 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:33.174534 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:35.243498 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:35.243463 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:35.244059 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:35.243800 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:35.256122 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:35.256102 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:35.635259 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:35.635226 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:07:56.124111 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.124076 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5"] Jul 22 01:07:56.128620 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.128570 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5"] Jul 22 01:07:56.128813 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.128664 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.131502 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.131476 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-75fe9f\"/\"openshift-service-ca.crt\"" Jul 22 01:07:56.132572 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.132550 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-75fe9f\"/\"test-model-75fe9f-kserve-self-signed-certs\"" Jul 22 01:07:56.132796 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.132768 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-75fe9f\"/\"kube-root-ca.crt\"" Jul 22 01:07:56.132904 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.132884 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-75fe9f\"/\"default-dockercfg-6scmw\"" Jul 22 01:07:56.147388 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.147366 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:07:56.255077 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.255054 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-model-cache\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.255180 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.255087 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e1f033d-2325-48fc-adaa-a467c2c332ba-tls-certs\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.255180 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.255164 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-home\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.255281 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.255218 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-tmp-dir\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.255281 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.255264 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-dshm\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.255368 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.255349 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt8pv\" (UniqueName: \"kubernetes.io/projected/2e1f033d-2325-48fc-adaa-a467c2c332ba-kube-api-access-bt8pv\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.355953 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.355925 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-model-cache\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.356079 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.355958 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e1f033d-2325-48fc-adaa-a467c2c332ba-tls-certs\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.356079 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.355999 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-home\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.356079 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.356017 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-tmp-dir\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.356079 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.356039 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-dshm\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.356079 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.356076 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt8pv\" (UniqueName: \"kubernetes.io/projected/2e1f033d-2325-48fc-adaa-a467c2c332ba-kube-api-access-bt8pv\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.356401 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.356369 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-model-cache\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.356401 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.356388 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-tmp-dir\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.356586 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.356514 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-home\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.358467 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.358447 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-dshm\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.358914 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.358893 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e1f033d-2325-48fc-adaa-a467c2c332ba-tls-certs\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.366142 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.366110 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt8pv\" (UniqueName: \"kubernetes.io/projected/2e1f033d-2325-48fc-adaa-a467c2c332ba-kube-api-access-bt8pv\") pod \"test-model-75fe9f-kserve-5f7f95f57c-87zm5\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.439199 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.439140 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:07:56.573756 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.573726 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5"] Jul 22 01:07:56.575579 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:07:56.575550 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e1f033d_2325_48fc_adaa_a467c2c332ba.slice/crio-7179ce9f57f5439afa0827d71625a59dd987ff8579a00d9d92cbd610de865dda WatchSource:0}: Error finding container 7179ce9f57f5439afa0827d71625a59dd987ff8579a00d9d92cbd610de865dda: Status 404 returned error can't find the container with id 7179ce9f57f5439afa0827d71625a59dd987ff8579a00d9d92cbd610de865dda Jul 22 01:07:56.730338 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.730263 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" event={"ID":"2e1f033d-2325-48fc-adaa-a467c2c332ba","Type":"ContainerStarted","Data":"e79acfb8bee51576ea10f34baa7079c6ddb4b12830e494d8ca9a4407c447fcfd"} Jul 22 01:07:56.730338 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.730307 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" event={"ID":"2e1f033d-2325-48fc-adaa-a467c2c332ba","Type":"ContainerStarted","Data":"7179ce9f57f5439afa0827d71625a59dd987ff8579a00d9d92cbd610de865dda"} Jul 22 01:07:56.749965 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:07:56.749921 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" podStartSLOduration=0.749906216 podStartE2EDuration="749.906216ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:07:56.748419011 +0000 UTC m=+2485.074491213" watchObservedRunningTime="2026-07-22 01:07:56.749906216 +0000 UTC m=+2485.075978418" Jul 22 01:08:06.439499 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:06.439452 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:08:06.440153 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:06.439600 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:08:06.452024 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:06.452001 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:08:06.793971 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:06.793942 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:08:35.622210 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:35.622171 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h"] Jul 22 01:08:36.568205 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:36.568167 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5"] Jul 22 01:08:37.852812 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:37.852754 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:08:41.620931 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:41.620895 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:08:41.675755 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:41.675727 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:08:47.282130 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:47.282100 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:08:50.643645 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:50.643583 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" podUID="8c0a77c2-f94a-438b-9b26-52b9678b1fa8" containerName="main" containerID="cri-o://90ebf297467e4244108a96251964d6774ad5314e774d1731de038cab8b88d750" gracePeriod=45 Jul 22 01:08:50.894185 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:50.894130 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:08:50.993556 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:50.993521 2567 generic.go:356] "Generic (PLEG): container finished" podID="8c0a77c2-f94a-438b-9b26-52b9678b1fa8" containerID="90ebf297467e4244108a96251964d6774ad5314e774d1731de038cab8b88d750" exitCode=0 Jul 22 01:08:50.993711 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:50.993559 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" Jul 22 01:08:50.993711 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:50.993572 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" event={"ID":"8c0a77c2-f94a-438b-9b26-52b9678b1fa8","Type":"ContainerDied","Data":"90ebf297467e4244108a96251964d6774ad5314e774d1731de038cab8b88d750"} Jul 22 01:08:50.993711 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:50.993603 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h" event={"ID":"8c0a77c2-f94a-438b-9b26-52b9678b1fa8","Type":"ContainerDied","Data":"c404adf4fa964e270c7112aa98c49497247d2628844c366fe26cbb5f9fcf4f39"} Jul 22 01:08:50.993711 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:50.993626 2567 scope.go:117] "RemoveContainer" containerID="90ebf297467e4244108a96251964d6774ad5314e774d1731de038cab8b88d750" Jul 22 01:08:51.001976 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.001955 2567 scope.go:117] "RemoveContainer" containerID="90ebf297467e4244108a96251964d6774ad5314e774d1731de038cab8b88d750" Jul 22 01:08:51.002151 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.002136 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-model-cache\") pod \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " Jul 22 01:08:51.002204 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.002175 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-tmp-dir\") pod \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " Jul 22 01:08:51.002244 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.002237 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bnzs\" (UniqueName: \"kubernetes.io/projected/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-kube-api-access-6bnzs\") pod \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " Jul 22 01:08:51.002291 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.002260 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-home\") pod \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " Jul 22 01:08:51.002291 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.002285 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-dshm\") pod \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " Jul 22 01:08:51.002392 ip-10-0-132-178 kubenswrapper[2567]: E0722 01:08:51.002337 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ebf297467e4244108a96251964d6774ad5314e774d1731de038cab8b88d750\": container with ID starting with 90ebf297467e4244108a96251964d6774ad5314e774d1731de038cab8b88d750 not found: ID does not exist" containerID="90ebf297467e4244108a96251964d6774ad5314e774d1731de038cab8b88d750" Jul 22 01:08:51.002392 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.002364 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c0a77c2-f94a-438b-9b26-52b9678b1fa8" (UID: "8c0a77c2-f94a-438b-9b26-52b9678b1fa8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:08:51.002392 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.002372 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ebf297467e4244108a96251964d6774ad5314e774d1731de038cab8b88d750"} err="failed to get container status \"90ebf297467e4244108a96251964d6774ad5314e774d1731de038cab8b88d750\": rpc error: code = NotFound desc = could not find container \"90ebf297467e4244108a96251964d6774ad5314e774d1731de038cab8b88d750\": container with ID starting with 90ebf297467e4244108a96251964d6774ad5314e774d1731de038cab8b88d750 not found: ID does not exist" Jul 22 01:08:51.002392 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.002374 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-tls-certs\") pod \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\" (UID: \"8c0a77c2-f94a-438b-9b26-52b9678b1fa8\") " Jul 22 01:08:51.002613 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.002469 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c0a77c2-f94a-438b-9b26-52b9678b1fa8" (UID: "8c0a77c2-f94a-438b-9b26-52b9678b1fa8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:08:51.002670 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.002636 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-home" (OuterVolumeSpecName: "home") pod "8c0a77c2-f94a-438b-9b26-52b9678b1fa8" (UID: "8c0a77c2-f94a-438b-9b26-52b9678b1fa8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:08:51.002732 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.002714 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-home\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:08:51.002797 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.002738 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-model-cache\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:08:51.002797 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.002752 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-tmp-dir\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:08:51.004491 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.004469 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c0a77c2-f94a-438b-9b26-52b9678b1fa8" (UID: "8c0a77c2-f94a-438b-9b26-52b9678b1fa8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:08:51.004577 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.004513 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-dshm" (OuterVolumeSpecName: "dshm") pod "8c0a77c2-f94a-438b-9b26-52b9678b1fa8" (UID: "8c0a77c2-f94a-438b-9b26-52b9678b1fa8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:08:51.004577 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.004555 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-kube-api-access-6bnzs" (OuterVolumeSpecName: "kube-api-access-6bnzs") pod "8c0a77c2-f94a-438b-9b26-52b9678b1fa8" (UID: "8c0a77c2-f94a-438b-9b26-52b9678b1fa8"). InnerVolumeSpecName "kube-api-access-6bnzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:08:51.103720 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.103695 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bnzs\" (UniqueName: \"kubernetes.io/projected/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-kube-api-access-6bnzs\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:08:51.103720 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.103719 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-dshm\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:08:51.103878 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.103733 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c0a77c2-f94a-438b-9b26-52b9678b1fa8-tls-certs\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:08:51.315947 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.315921 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h"] Jul 22 01:08:51.318900 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.318879 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-805a5b/test-model-805a5b-kserve-647f747547-8p49h"] Jul 22 01:08:51.588112 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.588051 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" podUID="2e1f033d-2325-48fc-adaa-a467c2c332ba" containerName="main" containerID="cri-o://e79acfb8bee51576ea10f34baa7079c6ddb4b12830e494d8ca9a4407c447fcfd" gracePeriod=45 Jul 22 01:08:51.776079 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.776037 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:08:51.834188 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.834168 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:08:51.910636 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.910576 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt8pv\" (UniqueName: \"kubernetes.io/projected/2e1f033d-2325-48fc-adaa-a467c2c332ba-kube-api-access-bt8pv\") pod \"2e1f033d-2325-48fc-adaa-a467c2c332ba\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " Jul 22 01:08:51.910748 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.910644 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-dshm\") pod \"2e1f033d-2325-48fc-adaa-a467c2c332ba\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " Jul 22 01:08:51.910748 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.910669 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-model-cache\") pod \"2e1f033d-2325-48fc-adaa-a467c2c332ba\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " Jul 22 01:08:51.910748 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.910687 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-tmp-dir\") pod \"2e1f033d-2325-48fc-adaa-a467c2c332ba\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " Jul 22 01:08:51.910748 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.910742 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e1f033d-2325-48fc-adaa-a467c2c332ba-tls-certs\") pod \"2e1f033d-2325-48fc-adaa-a467c2c332ba\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " Jul 22 01:08:51.910969 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.910765 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-home\") pod \"2e1f033d-2325-48fc-adaa-a467c2c332ba\" (UID: \"2e1f033d-2325-48fc-adaa-a467c2c332ba\") " Jul 22 01:08:51.911029 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.910983 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e1f033d-2325-48fc-adaa-a467c2c332ba" (UID: "2e1f033d-2325-48fc-adaa-a467c2c332ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:08:51.911029 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.911000 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e1f033d-2325-48fc-adaa-a467c2c332ba" (UID: "2e1f033d-2325-48fc-adaa-a467c2c332ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:08:51.911136 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.911096 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-model-cache\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:08:51.911136 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.911111 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-tmp-dir\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:08:51.911222 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.911128 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-home" (OuterVolumeSpecName: "home") pod "2e1f033d-2325-48fc-adaa-a467c2c332ba" (UID: "2e1f033d-2325-48fc-adaa-a467c2c332ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:08:51.912901 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.912871 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1f033d-2325-48fc-adaa-a467c2c332ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e1f033d-2325-48fc-adaa-a467c2c332ba" (UID: "2e1f033d-2325-48fc-adaa-a467c2c332ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:08:51.913196 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.913170 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-dshm" (OuterVolumeSpecName: "dshm") pod "2e1f033d-2325-48fc-adaa-a467c2c332ba" (UID: "2e1f033d-2325-48fc-adaa-a467c2c332ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:08:51.913196 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.913174 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1f033d-2325-48fc-adaa-a467c2c332ba-kube-api-access-bt8pv" (OuterVolumeSpecName: "kube-api-access-bt8pv") pod "2e1f033d-2325-48fc-adaa-a467c2c332ba" (UID: "2e1f033d-2325-48fc-adaa-a467c2c332ba"). InnerVolumeSpecName "kube-api-access-bt8pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:08:51.999117 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.999091 2567 generic.go:356] "Generic (PLEG): container finished" podID="2e1f033d-2325-48fc-adaa-a467c2c332ba" containerID="e79acfb8bee51576ea10f34baa7079c6ddb4b12830e494d8ca9a4407c447fcfd" exitCode=0 Jul 22 01:08:51.999221 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.999134 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" Jul 22 01:08:51.999221 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.999177 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" event={"ID":"2e1f033d-2325-48fc-adaa-a467c2c332ba","Type":"ContainerDied","Data":"e79acfb8bee51576ea10f34baa7079c6ddb4b12830e494d8ca9a4407c447fcfd"} Jul 22 01:08:51.999313 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.999221 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5" event={"ID":"2e1f033d-2325-48fc-adaa-a467c2c332ba","Type":"ContainerDied","Data":"7179ce9f57f5439afa0827d71625a59dd987ff8579a00d9d92cbd610de865dda"} Jul 22 01:08:51.999313 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:51.999251 2567 scope.go:117] "RemoveContainer" containerID="e79acfb8bee51576ea10f34baa7079c6ddb4b12830e494d8ca9a4407c447fcfd" Jul 22 01:08:52.007607 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:52.007591 2567 scope.go:117] "RemoveContainer" containerID="e79acfb8bee51576ea10f34baa7079c6ddb4b12830e494d8ca9a4407c447fcfd" Jul 22 01:08:52.007868 ip-10-0-132-178 kubenswrapper[2567]: E0722 01:08:52.007847 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e79acfb8bee51576ea10f34baa7079c6ddb4b12830e494d8ca9a4407c447fcfd\": container with ID starting with e79acfb8bee51576ea10f34baa7079c6ddb4b12830e494d8ca9a4407c447fcfd not found: ID does not exist" containerID="e79acfb8bee51576ea10f34baa7079c6ddb4b12830e494d8ca9a4407c447fcfd" Jul 22 01:08:52.007962 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:52.007872 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79acfb8bee51576ea10f34baa7079c6ddb4b12830e494d8ca9a4407c447fcfd"} err="failed to get container status \"e79acfb8bee51576ea10f34baa7079c6ddb4b12830e494d8ca9a4407c447fcfd\": rpc error: code = NotFound desc = could not find container \"e79acfb8bee51576ea10f34baa7079c6ddb4b12830e494d8ca9a4407c447fcfd\": container with ID starting with e79acfb8bee51576ea10f34baa7079c6ddb4b12830e494d8ca9a4407c447fcfd not found: ID does not exist" Jul 22 01:08:52.011566 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:52.011547 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e1f033d-2325-48fc-adaa-a467c2c332ba-tls-certs\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:08:52.011566 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:52.011567 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-home\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:08:52.011669 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:52.011576 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt8pv\" (UniqueName: \"kubernetes.io/projected/2e1f033d-2325-48fc-adaa-a467c2c332ba-kube-api-access-bt8pv\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:08:52.011669 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:52.011585 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e1f033d-2325-48fc-adaa-a467c2c332ba-dshm\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:08:52.020569 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:52.020550 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5"] Jul 22 01:08:52.023928 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:52.023909 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-75fe9f/test-model-75fe9f-kserve-5f7f95f57c-87zm5"] Jul 22 01:08:52.124916 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:52.124880 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e1f033d-2325-48fc-adaa-a467c2c332ba" path="/var/lib/kubelet/pods/2e1f033d-2325-48fc-adaa-a467c2c332ba/volumes" Jul 22 01:08:52.125269 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:52.125256 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c0a77c2-f94a-438b-9b26-52b9678b1fa8" path="/var/lib/kubelet/pods/8c0a77c2-f94a-438b-9b26-52b9678b1fa8/volumes" Jul 22 01:08:52.207195 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:52.207129 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:08:58.287421 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:58.287383 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:08:58.384527 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:58.384482 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:08:58.462583 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:58.462538 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:08:58.699612 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:58.699540 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:08:59.425886 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:59.425855 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:08:59.527017 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:08:59.526989 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:09.481520 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:09.481486 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:10.554365 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:10.554330 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:13.916860 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:13.916812 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:14.307855 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:14.307827 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:20.796228 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:20.796190 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:20.892182 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:20.892137 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:21.805031 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:21.804987 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:22.022201 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:22.022169 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:22.107992 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:22.107909 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:22.227426 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:22.227396 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:24.891833 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:24.891760 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:24.982887 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:24.982843 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:25.136440 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:25.136408 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:25.439752 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:25.439724 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:26.072561 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:26.072528 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:26.164013 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:26.163983 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:29.412146 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:29.412104 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:29.903051 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:29.903004 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:30.350417 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:30.350383 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:31.894864 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:31.894823 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:32.971035 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:32.970991 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:33.060744 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:33.060711 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:33.202334 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:33.202293 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:33.537841 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:33.537805 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:34.089908 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:34.089873 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:34.316278 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:34.316239 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:37.453379 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:37.453318 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:46.124391 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:46.124352 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:48.132875 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:48.132836 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:56.493827 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:56.493701 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:09:56.592867 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:09:56.592832 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:01.894986 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:01.894946 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:05.921567 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:05.921532 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:06.180831 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:06.180718 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:12.328172 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:12.328132 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:12.430579 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:12.430544 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:12.721860 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:12.721749 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:12.874303 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:12.874261 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:28.455478 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:28.455420 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:33.056493 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:33.056459 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:33.444678 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:33.444589 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:39.346604 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:39.346539 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:39.447219 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:39.447175 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:39.603753 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:39.603671 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:39.828668 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:39.828637 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:40.171555 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:40.171514 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:40.485261 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:40.485174 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:40.704697 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:40.704666 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:43.287153 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:43.287107 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:43.378267 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:43.378237 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:43.519139 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:43.519103 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:44.669375 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:44.669339 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:44.839610 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:44.839581 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:47.781411 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:47.781368 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:48.110855 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:48.110741 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:48.419123 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:48.419024 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:48.623030 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:48.622987 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:51.356330 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:51.356283 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:51.878325 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:51.878285 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:51.983073 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:51.983045 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:54.541940 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:54.541901 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:54.606640 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:54.606608 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:10:58.267819 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:10:58.267768 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:11:00.981476 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:00.981437 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:11:05.889311 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:05.889271 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:11:05.989663 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:05.989623 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:11:08.315061 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.315031 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf"] Jul 22 01:11:08.315431 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.315411 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e1f033d-2325-48fc-adaa-a467c2c332ba" containerName="main" Jul 22 01:11:08.315431 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.315423 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e1f033d-2325-48fc-adaa-a467c2c332ba" containerName="main" Jul 22 01:11:08.315510 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.315442 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c0a77c2-f94a-438b-9b26-52b9678b1fa8" containerName="main" Jul 22 01:11:08.315510 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.315447 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c0a77c2-f94a-438b-9b26-52b9678b1fa8" containerName="main" Jul 22 01:11:08.315510 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.315507 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c0a77c2-f94a-438b-9b26-52b9678b1fa8" containerName="main" Jul 22 01:11:08.315602 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.315515 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e1f033d-2325-48fc-adaa-a467c2c332ba" containerName="main" Jul 22 01:11:08.318608 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.318590 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.321468 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.321443 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e4b9ac\"/\"openshift-service-ca.crt\"" Jul 22 01:11:08.321619 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.321448 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e4b9ac\"/\"kube-root-ca.crt\"" Jul 22 01:11:08.322462 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.322439 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e4b9ac\"/\"default-dockercfg-c4cv2\"" Jul 22 01:11:08.322600 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.322501 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e4b9ac\"/\"auth-test-model-e4b9ac-kserve-self-signed-certs\"" Jul 22 01:11:08.332855 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.331207 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf"] Jul 22 01:11:08.337259 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.337236 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-home\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.337390 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.337269 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-model-cache\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.337390 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.337298 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-dshm\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.337390 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.337319 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c60863b-152b-4509-8b51-b8c880471579-tls-certs\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.337390 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.337359 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-tmp-dir\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.337554 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.337392 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wqqn\" (UniqueName: \"kubernetes.io/projected/0c60863b-152b-4509-8b51-b8c880471579-kube-api-access-2wqqn\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.437963 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.437937 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wqqn\" (UniqueName: \"kubernetes.io/projected/0c60863b-152b-4509-8b51-b8c880471579-kube-api-access-2wqqn\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.438104 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.438006 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-home\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.438104 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.438027 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-model-cache\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.438104 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.438056 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-dshm\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.438104 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.438073 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c60863b-152b-4509-8b51-b8c880471579-tls-certs\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.438104 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.438099 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-tmp-dir\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.438473 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.438450 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-model-cache\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.438519 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.438478 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-home\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.438561 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.438525 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-tmp-dir\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.440354 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.440334 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-dshm\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.440511 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.440495 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c60863b-152b-4509-8b51-b8c880471579-tls-certs\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.447249 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.447212 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:11:08.447806 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.447757 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wqqn\" (UniqueName: \"kubernetes.io/projected/0c60863b-152b-4509-8b51-b8c880471579-kube-api-access-2wqqn\") pod \"auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.628897 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.628842 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:08.765021 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:08.764994 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf"] Jul 22 01:11:08.767114 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:11:08.767083 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c60863b_152b_4509_8b51_b8c880471579.slice/crio-604e488ab4fcd494dd1b0f664d6d9b15a5528769a6dbf80f0c78792609476708 WatchSource:0}: Error finding container 604e488ab4fcd494dd1b0f664d6d9b15a5528769a6dbf80f0c78792609476708: Status 404 returned error can't find the container with id 604e488ab4fcd494dd1b0f664d6d9b15a5528769a6dbf80f0c78792609476708 Jul 22 01:11:09.688111 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:09.688069 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" event={"ID":"0c60863b-152b-4509-8b51-b8c880471579","Type":"ContainerStarted","Data":"ad40ec3b30b18369436ddebd26ce5e7f6ec31c3d355d3415d10c24c0bfa01048"} Jul 22 01:11:09.688473 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:09.688118 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" event={"ID":"0c60863b-152b-4509-8b51-b8c880471579","Type":"ContainerStarted","Data":"604e488ab4fcd494dd1b0f664d6d9b15a5528769a6dbf80f0c78792609476708"} Jul 22 01:11:09.705507 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:09.705463 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" podStartSLOduration=1.7054460599999999 podStartE2EDuration="1.70544606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:11:09.704941608 +0000 UTC m=+2678.031013811" watchObservedRunningTime="2026-07-22 01:11:09.70544606 +0000 UTC m=+2678.031518262" Jul 22 01:11:13.933217 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:13.933179 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:11:14.033251 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:14.033213 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:11:18.629150 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:18.629104 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:18.629665 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:18.629167 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:18.642327 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:18.642293 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:18.746065 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:18.746043 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:11:39.670602 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:39.670553 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:11:39.773344 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:39.773318 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:11:40.274316 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:40.274292 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 01:11:40.292748 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:40.292722 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 01:11:58.335934 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.335899 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55"] Jul 22 01:11:58.339382 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.339362 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.342885 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.342863 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f35c1\"/\"kube-root-ca.crt\"" Jul 22 01:11:58.343010 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.342865 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f35c1\"/\"default-dockercfg-z426z\"" Jul 22 01:11:58.343772 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.343755 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f35c1\"/\"openshift-service-ca.crt\"" Jul 22 01:11:58.344056 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.344043 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f35c1\"/\"auth-test-model-4f35c1-kserve-self-signed-certs\"" Jul 22 01:11:58.350614 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.350595 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55"] Jul 22 01:11:58.371890 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.371869 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-tmp-dir\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.372002 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.371902 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-model-cache\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.372002 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.371929 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-home\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.372002 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.371946 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed472506-4d8f-4798-8f3e-2eb224d491c2-tls-certs\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.372002 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.371964 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-dshm\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.372192 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.372043 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvq56\" (UniqueName: \"kubernetes.io/projected/ed472506-4d8f-4798-8f3e-2eb224d491c2-kube-api-access-vvq56\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.374490 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.374465 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:11:58.472952 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.472926 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-tmp-dir\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.473048 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.472959 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-model-cache\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.473048 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.472986 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-home\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.473048 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.473004 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed472506-4d8f-4798-8f3e-2eb224d491c2-tls-certs\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.473048 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.473028 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-dshm\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.473218 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.473060 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvq56\" (UniqueName: \"kubernetes.io/projected/ed472506-4d8f-4798-8f3e-2eb224d491c2-kube-api-access-vvq56\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.473382 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.473350 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-tmp-dir\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.473434 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.473378 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-model-cache\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.473475 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.473442 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-home\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.475282 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.475262 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-dshm\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.475476 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.475459 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed472506-4d8f-4798-8f3e-2eb224d491c2-tls-certs\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.482598 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.482578 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvq56\" (UniqueName: \"kubernetes.io/projected/ed472506-4d8f-4798-8f3e-2eb224d491c2-kube-api-access-vvq56\") pod \"auth-test-model-4f35c1-kserve-df5d87cf5-2vf55\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.648015 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.647916 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:11:58.787181 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.787155 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55"] Jul 22 01:11:58.789128 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:11:58.789093 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded472506_4d8f_4798_8f3e_2eb224d491c2.slice/crio-52e4243039e37d9aa9004d53394cf185600e43f3ac146c95a6e737acc5e046d0 WatchSource:0}: Error finding container 52e4243039e37d9aa9004d53394cf185600e43f3ac146c95a6e737acc5e046d0: Status 404 returned error can't find the container with id 52e4243039e37d9aa9004d53394cf185600e43f3ac146c95a6e737acc5e046d0 Jul 22 01:11:58.941652 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.941570 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" event={"ID":"ed472506-4d8f-4798-8f3e-2eb224d491c2","Type":"ContainerStarted","Data":"00f9d08769800fc06f0b9de614cddc0d9ce5babbb8033fedefef7ba80f440ad5"} Jul 22 01:11:58.941652 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.941617 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" event={"ID":"ed472506-4d8f-4798-8f3e-2eb224d491c2","Type":"ContainerStarted","Data":"52e4243039e37d9aa9004d53394cf185600e43f3ac146c95a6e737acc5e046d0"} Jul 22 01:11:58.961003 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:11:58.960960 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" podStartSLOduration=0.960944305 podStartE2EDuration="960.944305ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:11:58.959059891 +0000 UTC m=+2727.285132114" watchObservedRunningTime="2026-07-22 01:11:58.960944305 +0000 UTC m=+2727.287016507" Jul 22 01:12:08.648837 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:12:08.648770 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:12:08.649344 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:12:08.648853 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:12:08.661533 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:12:08.661501 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:12:09.008091 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:12:09.008011 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:13:51.331197 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:13:51.331150 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf"] Jul 22 01:13:51.376804 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:13:51.376754 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:13:52.145536 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:13:52.145497 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55"] Jul 22 01:13:54.425370 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:13:54.425337 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:14:06.350654 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.350590 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" podUID="0c60863b-152b-4509-8b51-b8c880471579" containerName="main" containerID="cri-o://ad40ec3b30b18369436ddebd26ce5e7f6ec31c3d355d3415d10c24c0bfa01048" gracePeriod=45 Jul 22 01:14:06.596997 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.596966 2567 generic.go:356] "Generic (PLEG): container finished" podID="0c60863b-152b-4509-8b51-b8c880471579" containerID="ad40ec3b30b18369436ddebd26ce5e7f6ec31c3d355d3415d10c24c0bfa01048" exitCode=0 Jul 22 01:14:06.597112 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.597036 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" event={"ID":"0c60863b-152b-4509-8b51-b8c880471579","Type":"ContainerDied","Data":"ad40ec3b30b18369436ddebd26ce5e7f6ec31c3d355d3415d10c24c0bfa01048"} Jul 22 01:14:06.597112 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.597077 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" event={"ID":"0c60863b-152b-4509-8b51-b8c880471579","Type":"ContainerDied","Data":"604e488ab4fcd494dd1b0f664d6d9b15a5528769a6dbf80f0c78792609476708"} Jul 22 01:14:06.597112 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.597088 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="604e488ab4fcd494dd1b0f664d6d9b15a5528769a6dbf80f0c78792609476708" Jul 22 01:14:06.607029 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.606968 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:14:06.769463 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.769430 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-home\") pod \"0c60863b-152b-4509-8b51-b8c880471579\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " Jul 22 01:14:06.769652 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.769474 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wqqn\" (UniqueName: \"kubernetes.io/projected/0c60863b-152b-4509-8b51-b8c880471579-kube-api-access-2wqqn\") pod \"0c60863b-152b-4509-8b51-b8c880471579\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " Jul 22 01:14:06.769652 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.769498 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c60863b-152b-4509-8b51-b8c880471579-tls-certs\") pod \"0c60863b-152b-4509-8b51-b8c880471579\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " Jul 22 01:14:06.769652 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.769520 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-dshm\") pod \"0c60863b-152b-4509-8b51-b8c880471579\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " Jul 22 01:14:06.769652 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.769548 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-model-cache\") pod \"0c60863b-152b-4509-8b51-b8c880471579\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " Jul 22 01:14:06.769652 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.769595 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-tmp-dir\") pod \"0c60863b-152b-4509-8b51-b8c880471579\" (UID: \"0c60863b-152b-4509-8b51-b8c880471579\") " Jul 22 01:14:06.769938 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.769808 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-home" (OuterVolumeSpecName: "home") pod "0c60863b-152b-4509-8b51-b8c880471579" (UID: "0c60863b-152b-4509-8b51-b8c880471579"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:14:06.769938 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.769915 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c60863b-152b-4509-8b51-b8c880471579" (UID: "0c60863b-152b-4509-8b51-b8c880471579"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:14:06.770049 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.769961 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c60863b-152b-4509-8b51-b8c880471579" (UID: "0c60863b-152b-4509-8b51-b8c880471579"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:14:06.771866 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.771839 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c60863b-152b-4509-8b51-b8c880471579-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c60863b-152b-4509-8b51-b8c880471579" (UID: "0c60863b-152b-4509-8b51-b8c880471579"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:14:06.771866 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.771850 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-dshm" (OuterVolumeSpecName: "dshm") pod "0c60863b-152b-4509-8b51-b8c880471579" (UID: "0c60863b-152b-4509-8b51-b8c880471579"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:14:06.771997 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.771928 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c60863b-152b-4509-8b51-b8c880471579-kube-api-access-2wqqn" (OuterVolumeSpecName: "kube-api-access-2wqqn") pod "0c60863b-152b-4509-8b51-b8c880471579" (UID: "0c60863b-152b-4509-8b51-b8c880471579"). InnerVolumeSpecName "kube-api-access-2wqqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:14:06.870605 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.870506 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-home\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:14:06.870605 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.870543 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wqqn\" (UniqueName: \"kubernetes.io/projected/0c60863b-152b-4509-8b51-b8c880471579-kube-api-access-2wqqn\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:14:06.870605 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.870557 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c60863b-152b-4509-8b51-b8c880471579-tls-certs\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:14:06.870605 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.870569 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-dshm\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:14:06.870605 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.870579 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-model-cache\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:14:06.870605 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:06.870590 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c60863b-152b-4509-8b51-b8c880471579-tmp-dir\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:14:07.165480 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.165392 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" podUID="ed472506-4d8f-4798-8f3e-2eb224d491c2" containerName="main" containerID="cri-o://00f9d08769800fc06f0b9de614cddc0d9ce5babbb8033fedefef7ba80f440ad5" gracePeriod=45 Jul 22 01:14:07.418845 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.418767 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:14:07.576422 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.576386 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed472506-4d8f-4798-8f3e-2eb224d491c2-tls-certs\") pod \"ed472506-4d8f-4798-8f3e-2eb224d491c2\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " Jul 22 01:14:07.576611 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.576446 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvq56\" (UniqueName: \"kubernetes.io/projected/ed472506-4d8f-4798-8f3e-2eb224d491c2-kube-api-access-vvq56\") pod \"ed472506-4d8f-4798-8f3e-2eb224d491c2\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " Jul 22 01:14:07.576611 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.576511 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-model-cache\") pod \"ed472506-4d8f-4798-8f3e-2eb224d491c2\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " Jul 22 01:14:07.576611 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.576532 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-home\") pod \"ed472506-4d8f-4798-8f3e-2eb224d491c2\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " Jul 22 01:14:07.576611 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.576603 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-dshm\") pod \"ed472506-4d8f-4798-8f3e-2eb224d491c2\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " Jul 22 01:14:07.576869 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.576654 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-tmp-dir\") pod \"ed472506-4d8f-4798-8f3e-2eb224d491c2\" (UID: \"ed472506-4d8f-4798-8f3e-2eb224d491c2\") " Jul 22 01:14:07.576869 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.576745 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed472506-4d8f-4798-8f3e-2eb224d491c2" (UID: "ed472506-4d8f-4798-8f3e-2eb224d491c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:14:07.576971 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.576862 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-home" (OuterVolumeSpecName: "home") pod "ed472506-4d8f-4798-8f3e-2eb224d491c2" (UID: "ed472506-4d8f-4798-8f3e-2eb224d491c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:14:07.577013 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.576983 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed472506-4d8f-4798-8f3e-2eb224d491c2" (UID: "ed472506-4d8f-4798-8f3e-2eb224d491c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:14:07.577090 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.577077 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-model-cache\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:14:07.577126 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.577093 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-home\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:14:07.577126 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.577103 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-tmp-dir\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:14:07.578787 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.578756 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-dshm" (OuterVolumeSpecName: "dshm") pod "ed472506-4d8f-4798-8f3e-2eb224d491c2" (UID: "ed472506-4d8f-4798-8f3e-2eb224d491c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:14:07.579047 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.579025 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed472506-4d8f-4798-8f3e-2eb224d491c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed472506-4d8f-4798-8f3e-2eb224d491c2" (UID: "ed472506-4d8f-4798-8f3e-2eb224d491c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:14:07.579145 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.579060 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed472506-4d8f-4798-8f3e-2eb224d491c2-kube-api-access-vvq56" (OuterVolumeSpecName: "kube-api-access-vvq56") pod "ed472506-4d8f-4798-8f3e-2eb224d491c2" (UID: "ed472506-4d8f-4798-8f3e-2eb224d491c2"). InnerVolumeSpecName "kube-api-access-vvq56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:14:07.602771 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.602743 2567 generic.go:356] "Generic (PLEG): container finished" podID="ed472506-4d8f-4798-8f3e-2eb224d491c2" containerID="00f9d08769800fc06f0b9de614cddc0d9ce5babbb8033fedefef7ba80f440ad5" exitCode=0 Jul 22 01:14:07.602912 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.602807 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" Jul 22 01:14:07.602912 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.602823 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf" Jul 22 01:14:07.602912 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.602833 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" event={"ID":"ed472506-4d8f-4798-8f3e-2eb224d491c2","Type":"ContainerDied","Data":"00f9d08769800fc06f0b9de614cddc0d9ce5babbb8033fedefef7ba80f440ad5"} Jul 22 01:14:07.602912 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.602875 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55" event={"ID":"ed472506-4d8f-4798-8f3e-2eb224d491c2","Type":"ContainerDied","Data":"52e4243039e37d9aa9004d53394cf185600e43f3ac146c95a6e737acc5e046d0"} Jul 22 01:14:07.602912 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.602892 2567 scope.go:117] "RemoveContainer" containerID="00f9d08769800fc06f0b9de614cddc0d9ce5babbb8033fedefef7ba80f440ad5" Jul 22 01:14:07.612980 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.612956 2567 scope.go:117] "RemoveContainer" containerID="00f9d08769800fc06f0b9de614cddc0d9ce5babbb8033fedefef7ba80f440ad5" Jul 22 01:14:07.613247 ip-10-0-132-178 kubenswrapper[2567]: E0722 01:14:07.613222 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f9d08769800fc06f0b9de614cddc0d9ce5babbb8033fedefef7ba80f440ad5\": container with ID starting with 00f9d08769800fc06f0b9de614cddc0d9ce5babbb8033fedefef7ba80f440ad5 not found: ID does not exist" containerID="00f9d08769800fc06f0b9de614cddc0d9ce5babbb8033fedefef7ba80f440ad5" Jul 22 01:14:07.613314 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.613256 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f9d08769800fc06f0b9de614cddc0d9ce5babbb8033fedefef7ba80f440ad5"} err="failed to get container status \"00f9d08769800fc06f0b9de614cddc0d9ce5babbb8033fedefef7ba80f440ad5\": rpc error: code = NotFound desc = could not find container \"00f9d08769800fc06f0b9de614cddc0d9ce5babbb8033fedefef7ba80f440ad5\": container with ID starting with 00f9d08769800fc06f0b9de614cddc0d9ce5babbb8033fedefef7ba80f440ad5 not found: ID does not exist" Jul 22 01:14:07.626400 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.626377 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55"] Jul 22 01:14:07.630368 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.630350 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/auth-test-model-4f35c1-kserve-df5d87cf5-2vf55"] Jul 22 01:14:07.641517 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.641497 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf"] Jul 22 01:14:07.645245 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.645219 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/auth-test-model-e4b9ac-kserve-6fc7d7f9c9-bddqf"] Jul 22 01:14:07.678145 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.678079 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed472506-4d8f-4798-8f3e-2eb224d491c2-dshm\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:14:07.678145 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.678104 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed472506-4d8f-4798-8f3e-2eb224d491c2-tls-certs\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:14:07.678145 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:07.678115 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvq56\" (UniqueName: \"kubernetes.io/projected/ed472506-4d8f-4798-8f3e-2eb224d491c2-kube-api-access-vvq56\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:14:08.122684 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:08.121720 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c60863b-152b-4509-8b51-b8c880471579" path="/var/lib/kubelet/pods/0c60863b-152b-4509-8b51-b8c880471579/volumes" Jul 22 01:14:08.122684 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:08.122361 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed472506-4d8f-4798-8f3e-2eb224d491c2" path="/var/lib/kubelet/pods/ed472506-4d8f-4798-8f3e-2eb224d491c2/volumes" Jul 22 01:14:17.011059 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.011029 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl"] Jul 22 01:14:17.011435 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.011402 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c60863b-152b-4509-8b51-b8c880471579" containerName="main" Jul 22 01:14:17.011435 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.011414 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c60863b-152b-4509-8b51-b8c880471579" containerName="main" Jul 22 01:14:17.011435 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.011436 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed472506-4d8f-4798-8f3e-2eb224d491c2" containerName="main" Jul 22 01:14:17.011539 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.011441 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed472506-4d8f-4798-8f3e-2eb224d491c2" containerName="main" Jul 22 01:14:17.011539 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.011506 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c60863b-152b-4509-8b51-b8c880471579" containerName="main" Jul 22 01:14:17.011539 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.011514 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed472506-4d8f-4798-8f3e-2eb224d491c2" containerName="main" Jul 22 01:14:17.014505 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.014486 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.018858 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.018828 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e4b9ac\"/\"kube-root-ca.crt\"" Jul 22 01:14:17.018858 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.018845 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e4b9ac\"/\"openshift-service-ca.crt\"" Jul 22 01:14:17.018858 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.018851 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e4b9ac\"/\"sub-test-model-e4b9ac-kserve-self-signed-certs\"" Jul 22 01:14:17.019096 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.018858 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e4b9ac\"/\"default-dockercfg-c4cv2\"" Jul 22 01:14:17.024625 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.024599 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl"] Jul 22 01:14:17.164681 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.164657 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-model-cache\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.164816 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.164695 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-dshm\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.164816 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.164714 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-tmp-dir\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.164915 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.164828 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvdsl\" (UniqueName: \"kubernetes.io/projected/f06b6291-4f37-46ab-9a49-9ae3bcea3167-kube-api-access-vvdsl\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.164915 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.164876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f06b6291-4f37-46ab-9a49-9ae3bcea3167-tls-certs\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.165002 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.164922 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-home\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.265635 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.265568 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvdsl\" (UniqueName: \"kubernetes.io/projected/f06b6291-4f37-46ab-9a49-9ae3bcea3167-kube-api-access-vvdsl\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.265635 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.265612 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f06b6291-4f37-46ab-9a49-9ae3bcea3167-tls-certs\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.265839 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.265646 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-home\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.265839 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.265683 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-model-cache\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.265839 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.265716 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-dshm\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.265839 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.265740 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-tmp-dir\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.266180 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.266160 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-tmp-dir\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.266232 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.266212 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-model-cache\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.266277 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.266241 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-home\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.268043 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.268016 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-dshm\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.268292 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.268276 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f06b6291-4f37-46ab-9a49-9ae3bcea3167-tls-certs\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.273773 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.273750 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvdsl\" (UniqueName: \"kubernetes.io/projected/f06b6291-4f37-46ab-9a49-9ae3bcea3167-kube-api-access-vvdsl\") pod \"sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.325046 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.325025 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:17.452312 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.452288 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl"] Jul 22 01:14:17.454683 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:14:17.454649 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf06b6291_4f37_46ab_9a49_9ae3bcea3167.slice/crio-31376c6ca4ce88494d338bfaa903714a461736a02daf17fc22f4082a48aab9ab WatchSource:0}: Error finding container 31376c6ca4ce88494d338bfaa903714a461736a02daf17fc22f4082a48aab9ab: Status 404 returned error can't find the container with id 31376c6ca4ce88494d338bfaa903714a461736a02daf17fc22f4082a48aab9ab Jul 22 01:14:17.651175 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.651096 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" event={"ID":"f06b6291-4f37-46ab-9a49-9ae3bcea3167","Type":"ContainerStarted","Data":"90521afea1396f5563a6d4abf5899726bef1385e8ca274cff1039594fbb2ef21"} Jul 22 01:14:17.651175 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.651136 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" event={"ID":"f06b6291-4f37-46ab-9a49-9ae3bcea3167","Type":"ContainerStarted","Data":"31376c6ca4ce88494d338bfaa903714a461736a02daf17fc22f4082a48aab9ab"} Jul 22 01:14:17.657715 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.657690 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:14:17.667822 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:17.667765 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" podStartSLOduration=1.6677460370000001 podStartE2EDuration="1.667746037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:14:17.666829519 +0000 UTC m=+2865.992901718" watchObservedRunningTime="2026-07-22 01:14:17.667746037 +0000 UTC m=+2865.993818240" Jul 22 01:14:27.325982 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:27.325893 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:27.325982 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:27.325940 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:27.338547 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:27.338522 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:27.718187 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:27.718112 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:14:29.831645 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:29.831616 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:14:45.011894 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.011859 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k"] Jul 22 01:14:45.015446 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.015422 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.019231 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.019209 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f35c1\"/\"default-dockercfg-z426z\"" Jul 22 01:14:45.019231 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.019229 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f35c1\"/\"sub-test-model-4f35c1-kserve-self-signed-certs\"" Jul 22 01:14:45.019430 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.019268 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f35c1\"/\"openshift-service-ca.crt\"" Jul 22 01:14:45.019430 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.019352 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f35c1\"/\"kube-root-ca.crt\"" Jul 22 01:14:45.024541 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.024517 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k"] Jul 22 01:14:45.042960 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.042936 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:14:45.127756 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.127729 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccfd9d9-bc15-4297-ac36-2670eae62415-tls-certs\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.127912 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.127806 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9v5c\" (UniqueName: \"kubernetes.io/projected/4ccfd9d9-bc15-4297-ac36-2670eae62415-kube-api-access-s9v5c\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.127912 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.127841 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-home\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.127912 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.127881 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-model-cache\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.128041 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.127945 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-tmp-dir\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.128041 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.127967 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-dshm\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.228463 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.228433 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-tmp-dir\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.228609 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.228466 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-dshm\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.228609 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.228495 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccfd9d9-bc15-4297-ac36-2670eae62415-tls-certs\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.228609 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.228570 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9v5c\" (UniqueName: \"kubernetes.io/projected/4ccfd9d9-bc15-4297-ac36-2670eae62415-kube-api-access-s9v5c\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.228804 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.228614 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-home\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.228804 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.228678 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-model-cache\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.229076 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.229049 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-home\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.229167 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.229096 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-tmp-dir\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.229167 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.229129 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-model-cache\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.230715 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.230685 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-dshm\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.231062 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.231043 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccfd9d9-bc15-4297-ac36-2670eae62415-tls-certs\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.237597 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.237576 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9v5c\" (UniqueName: \"kubernetes.io/projected/4ccfd9d9-bc15-4297-ac36-2670eae62415-kube-api-access-s9v5c\") pod \"sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.326894 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.326817 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:45.459707 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.459681 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k"] Jul 22 01:14:45.464470 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:14:45.464438 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ccfd9d9_bc15_4297_ac36_2670eae62415.slice/crio-53de2f2ce301e9903e639d0c523153357869dc018262d2dc6eba0cdece3e4685 WatchSource:0}: Error finding container 53de2f2ce301e9903e639d0c523153357869dc018262d2dc6eba0cdece3e4685: Status 404 returned error can't find the container with id 53de2f2ce301e9903e639d0c523153357869dc018262d2dc6eba0cdece3e4685 Jul 22 01:14:45.796983 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.796946 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" event={"ID":"4ccfd9d9-bc15-4297-ac36-2670eae62415","Type":"ContainerStarted","Data":"ab4db2c1585a4b9fe75fdab75b44e928fcac6fca107ed4f0e4691f6d196876b7"} Jul 22 01:14:45.797191 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.796996 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" event={"ID":"4ccfd9d9-bc15-4297-ac36-2670eae62415","Type":"ContainerStarted","Data":"53de2f2ce301e9903e639d0c523153357869dc018262d2dc6eba0cdece3e4685"} Jul 22 01:14:45.816697 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:45.816650 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" podStartSLOduration=1.81663376 podStartE2EDuration="1.81663376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:14:45.815249387 +0000 UTC m=+2894.141321589" watchObservedRunningTime="2026-07-22 01:14:45.81663376 +0000 UTC m=+2894.142705980" Jul 22 01:14:54.558670 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:54.558634 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:14:54.619059 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:54.619031 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:14:55.327545 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:55.327506 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:55.327719 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:55.327557 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:55.340390 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:55.340368 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:14:55.865433 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:14:55.865400 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:15:30.655246 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:30.655188 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:15:34.788914 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:34.788881 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl"] Jul 22 01:15:34.810381 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:34.810353 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:15:35.513235 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:35.513195 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k"] Jul 22 01:15:35.535493 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:35.535462 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:15:36.180156 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:36.180122 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:15:37.545366 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:37.545331 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:15:49.808767 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:49.808695 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" podUID="f06b6291-4f37-46ab-9a49-9ae3bcea3167" containerName="main" containerID="cri-o://90521afea1396f5563a6d4abf5899726bef1385e8ca274cff1039594fbb2ef21" gracePeriod=45 Jul 22 01:15:50.061572 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.061512 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:15:50.134490 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.134460 2567 generic.go:356] "Generic (PLEG): container finished" podID="f06b6291-4f37-46ab-9a49-9ae3bcea3167" containerID="90521afea1396f5563a6d4abf5899726bef1385e8ca274cff1039594fbb2ef21" exitCode=0 Jul 22 01:15:50.134615 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.134499 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" Jul 22 01:15:50.134615 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.134539 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" event={"ID":"f06b6291-4f37-46ab-9a49-9ae3bcea3167","Type":"ContainerDied","Data":"90521afea1396f5563a6d4abf5899726bef1385e8ca274cff1039594fbb2ef21"} Jul 22 01:15:50.134615 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.134575 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl" event={"ID":"f06b6291-4f37-46ab-9a49-9ae3bcea3167","Type":"ContainerDied","Data":"31376c6ca4ce88494d338bfaa903714a461736a02daf17fc22f4082a48aab9ab"} Jul 22 01:15:50.134615 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.134592 2567 scope.go:117] "RemoveContainer" containerID="90521afea1396f5563a6d4abf5899726bef1385e8ca274cff1039594fbb2ef21" Jul 22 01:15:50.143041 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.143025 2567 scope.go:117] "RemoveContainer" containerID="90521afea1396f5563a6d4abf5899726bef1385e8ca274cff1039594fbb2ef21" Jul 22 01:15:50.143306 ip-10-0-132-178 kubenswrapper[2567]: E0722 01:15:50.143285 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90521afea1396f5563a6d4abf5899726bef1385e8ca274cff1039594fbb2ef21\": container with ID starting with 90521afea1396f5563a6d4abf5899726bef1385e8ca274cff1039594fbb2ef21 not found: ID does not exist" containerID="90521afea1396f5563a6d4abf5899726bef1385e8ca274cff1039594fbb2ef21" Jul 22 01:15:50.143409 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.143312 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90521afea1396f5563a6d4abf5899726bef1385e8ca274cff1039594fbb2ef21"} err="failed to get container status \"90521afea1396f5563a6d4abf5899726bef1385e8ca274cff1039594fbb2ef21\": rpc error: code = NotFound desc = could not find container \"90521afea1396f5563a6d4abf5899726bef1385e8ca274cff1039594fbb2ef21\": container with ID starting with 90521afea1396f5563a6d4abf5899726bef1385e8ca274cff1039594fbb2ef21 not found: ID does not exist" Jul 22 01:15:50.161040 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.161020 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-home\") pod \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " Jul 22 01:15:50.161126 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.161055 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvdsl\" (UniqueName: \"kubernetes.io/projected/f06b6291-4f37-46ab-9a49-9ae3bcea3167-kube-api-access-vvdsl\") pod \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " Jul 22 01:15:50.161126 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.161085 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f06b6291-4f37-46ab-9a49-9ae3bcea3167-tls-certs\") pod \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " Jul 22 01:15:50.161126 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.161110 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-dshm\") pod \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " Jul 22 01:15:50.161291 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.161170 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-model-cache\") pod \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " Jul 22 01:15:50.161291 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.161193 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-tmp-dir\") pod \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\" (UID: \"f06b6291-4f37-46ab-9a49-9ae3bcea3167\") " Jul 22 01:15:50.161291 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.161258 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-home" (OuterVolumeSpecName: "home") pod "f06b6291-4f37-46ab-9a49-9ae3bcea3167" (UID: "f06b6291-4f37-46ab-9a49-9ae3bcea3167"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:15:50.161457 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.161430 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-model-cache" (OuterVolumeSpecName: "model-cache") pod "f06b6291-4f37-46ab-9a49-9ae3bcea3167" (UID: "f06b6291-4f37-46ab-9a49-9ae3bcea3167"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:15:50.161533 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.161455 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-home\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:15:50.161533 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.161459 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f06b6291-4f37-46ab-9a49-9ae3bcea3167" (UID: "f06b6291-4f37-46ab-9a49-9ae3bcea3167"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:15:50.163226 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.163202 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f06b6291-4f37-46ab-9a49-9ae3bcea3167-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f06b6291-4f37-46ab-9a49-9ae3bcea3167" (UID: "f06b6291-4f37-46ab-9a49-9ae3bcea3167"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:15:50.163226 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.163203 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06b6291-4f37-46ab-9a49-9ae3bcea3167-kube-api-access-vvdsl" (OuterVolumeSpecName: "kube-api-access-vvdsl") pod "f06b6291-4f37-46ab-9a49-9ae3bcea3167" (UID: "f06b6291-4f37-46ab-9a49-9ae3bcea3167"). InnerVolumeSpecName "kube-api-access-vvdsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:15:50.163357 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.163231 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-dshm" (OuterVolumeSpecName: "dshm") pod "f06b6291-4f37-46ab-9a49-9ae3bcea3167" (UID: "f06b6291-4f37-46ab-9a49-9ae3bcea3167"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:15:50.262759 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.262734 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-model-cache\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:15:50.262759 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.262754 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-tmp-dir\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:15:50.262759 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.262762 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvdsl\" (UniqueName: \"kubernetes.io/projected/f06b6291-4f37-46ab-9a49-9ae3bcea3167-kube-api-access-vvdsl\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:15:50.262928 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.262771 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f06b6291-4f37-46ab-9a49-9ae3bcea3167-tls-certs\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:15:50.262928 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.262804 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f06b6291-4f37-46ab-9a49-9ae3bcea3167-dshm\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:15:50.459546 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.459521 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl"] Jul 22 01:15:50.463516 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.463489 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/sub-test-model-e4b9ac-kserve-6799f5598c-rpvpl"] Jul 22 01:15:50.534029 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.533972 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" podUID="4ccfd9d9-bc15-4297-ac36-2670eae62415" containerName="main" containerID="cri-o://ab4db2c1585a4b9fe75fdab75b44e928fcac6fca107ed4f0e4691f6d196876b7" gracePeriod=45 Jul 22 01:15:50.770791 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.770757 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:15:50.868017 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.867980 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccfd9d9-bc15-4297-ac36-2670eae62415-tls-certs\") pod \"4ccfd9d9-bc15-4297-ac36-2670eae62415\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " Jul 22 01:15:50.868363 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.868031 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-model-cache\") pod \"4ccfd9d9-bc15-4297-ac36-2670eae62415\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " Jul 22 01:15:50.868363 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.868051 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9v5c\" (UniqueName: \"kubernetes.io/projected/4ccfd9d9-bc15-4297-ac36-2670eae62415-kube-api-access-s9v5c\") pod \"4ccfd9d9-bc15-4297-ac36-2670eae62415\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " Jul 22 01:15:50.868363 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.868076 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-dshm\") pod \"4ccfd9d9-bc15-4297-ac36-2670eae62415\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " Jul 22 01:15:50.868363 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.868101 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-home\") pod \"4ccfd9d9-bc15-4297-ac36-2670eae62415\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " Jul 22 01:15:50.868363 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.868129 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-tmp-dir\") pod \"4ccfd9d9-bc15-4297-ac36-2670eae62415\" (UID: \"4ccfd9d9-bc15-4297-ac36-2670eae62415\") " Jul 22 01:15:50.868363 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.868284 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ccfd9d9-bc15-4297-ac36-2670eae62415" (UID: "4ccfd9d9-bc15-4297-ac36-2670eae62415"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:15:50.868621 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.868370 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-home" (OuterVolumeSpecName: "home") pod "4ccfd9d9-bc15-4297-ac36-2670eae62415" (UID: "4ccfd9d9-bc15-4297-ac36-2670eae62415"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:15:50.868621 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.868445 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ccfd9d9-bc15-4297-ac36-2670eae62415" (UID: "4ccfd9d9-bc15-4297-ac36-2670eae62415"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:15:50.868621 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.868498 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-model-cache\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:15:50.868621 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.868517 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-home\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:15:50.870086 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.870067 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-dshm" (OuterVolumeSpecName: "dshm") pod "4ccfd9d9-bc15-4297-ac36-2670eae62415" (UID: "4ccfd9d9-bc15-4297-ac36-2670eae62415"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:15:50.870153 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.870103 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ccfd9d9-bc15-4297-ac36-2670eae62415-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ccfd9d9-bc15-4297-ac36-2670eae62415" (UID: "4ccfd9d9-bc15-4297-ac36-2670eae62415"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:15:50.870207 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.870190 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccfd9d9-bc15-4297-ac36-2670eae62415-kube-api-access-s9v5c" (OuterVolumeSpecName: "kube-api-access-s9v5c") pod "4ccfd9d9-bc15-4297-ac36-2670eae62415" (UID: "4ccfd9d9-bc15-4297-ac36-2670eae62415"). InnerVolumeSpecName "kube-api-access-s9v5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:15:50.969070 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.969021 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-tmp-dir\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:15:50.969070 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.969041 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccfd9d9-bc15-4297-ac36-2670eae62415-tls-certs\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:15:50.969070 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.969050 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9v5c\" (UniqueName: \"kubernetes.io/projected/4ccfd9d9-bc15-4297-ac36-2670eae62415-kube-api-access-s9v5c\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:15:50.969070 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:50.969058 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccfd9d9-bc15-4297-ac36-2670eae62415-dshm\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:15:51.141504 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:51.141477 2567 generic.go:356] "Generic (PLEG): container finished" podID="4ccfd9d9-bc15-4297-ac36-2670eae62415" containerID="ab4db2c1585a4b9fe75fdab75b44e928fcac6fca107ed4f0e4691f6d196876b7" exitCode=0 Jul 22 01:15:51.141665 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:51.141513 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" Jul 22 01:15:51.141665 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:51.141565 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" event={"ID":"4ccfd9d9-bc15-4297-ac36-2670eae62415","Type":"ContainerDied","Data":"ab4db2c1585a4b9fe75fdab75b44e928fcac6fca107ed4f0e4691f6d196876b7"} Jul 22 01:15:51.141665 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:51.141605 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k" event={"ID":"4ccfd9d9-bc15-4297-ac36-2670eae62415","Type":"ContainerDied","Data":"53de2f2ce301e9903e639d0c523153357869dc018262d2dc6eba0cdece3e4685"} Jul 22 01:15:51.141665 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:51.141627 2567 scope.go:117] "RemoveContainer" containerID="ab4db2c1585a4b9fe75fdab75b44e928fcac6fca107ed4f0e4691f6d196876b7" Jul 22 01:15:51.151111 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:51.151096 2567 scope.go:117] "RemoveContainer" containerID="ab4db2c1585a4b9fe75fdab75b44e928fcac6fca107ed4f0e4691f6d196876b7" Jul 22 01:15:51.151375 ip-10-0-132-178 kubenswrapper[2567]: E0722 01:15:51.151357 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4db2c1585a4b9fe75fdab75b44e928fcac6fca107ed4f0e4691f6d196876b7\": container with ID starting with ab4db2c1585a4b9fe75fdab75b44e928fcac6fca107ed4f0e4691f6d196876b7 not found: ID does not exist" containerID="ab4db2c1585a4b9fe75fdab75b44e928fcac6fca107ed4f0e4691f6d196876b7" Jul 22 01:15:51.151427 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:51.151385 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4db2c1585a4b9fe75fdab75b44e928fcac6fca107ed4f0e4691f6d196876b7"} err="failed to get container status \"ab4db2c1585a4b9fe75fdab75b44e928fcac6fca107ed4f0e4691f6d196876b7\": rpc error: code = NotFound desc = could not find container \"ab4db2c1585a4b9fe75fdab75b44e928fcac6fca107ed4f0e4691f6d196876b7\": container with ID starting with ab4db2c1585a4b9fe75fdab75b44e928fcac6fca107ed4f0e4691f6d196876b7 not found: ID does not exist" Jul 22 01:15:51.165234 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:51.165212 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k"] Jul 22 01:15:51.171278 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:51.171257 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/sub-test-model-4f35c1-kserve-6f59dd7755-n6p9k"] Jul 22 01:15:52.119610 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:52.119572 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ccfd9d9-bc15-4297-ac36-2670eae62415" path="/var/lib/kubelet/pods/4ccfd9d9-bc15-4297-ac36-2670eae62415/volumes" Jul 22 01:15:52.120025 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:15:52.119995 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f06b6291-4f37-46ab-9a49-9ae3bcea3167" path="/var/lib/kubelet/pods/f06b6291-4f37-46ab-9a49-9ae3bcea3167/volumes" Jul 22 01:16:02.466473 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:02.466437 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:16:06.668328 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:06.668294 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:16:06.783396 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:06.783354 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:16:07.715057 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.715025 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t"] Jul 22 01:16:07.715616 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.715587 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ccfd9d9-bc15-4297-ac36-2670eae62415" containerName="main" Jul 22 01:16:07.715616 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.715608 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ccfd9d9-bc15-4297-ac36-2670eae62415" containerName="main" Jul 22 01:16:07.715736 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.715647 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f06b6291-4f37-46ab-9a49-9ae3bcea3167" containerName="main" Jul 22 01:16:07.715736 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.715657 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f06b6291-4f37-46ab-9a49-9ae3bcea3167" containerName="main" Jul 22 01:16:07.715870 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.715751 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f06b6291-4f37-46ab-9a49-9ae3bcea3167" containerName="main" Jul 22 01:16:07.715870 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.715765 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ccfd9d9-bc15-4297-ac36-2670eae62415" containerName="main" Jul 22 01:16:07.720192 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.720168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.727349 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.727317 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f35c1\"/\"default-dockercfg-z426z\"" Jul 22 01:16:07.727443 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.727386 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f35c1\"/\"kube-root-ca.crt\"" Jul 22 01:16:07.727443 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.727409 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f35c1\"/\"openshift-service-ca.crt\"" Jul 22 01:16:07.727541 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.727432 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f35c1\"/\"rate-test-model-4f35c1-kserve-self-signed-certs\"" Jul 22 01:16:07.740479 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.740449 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t"] Jul 22 01:16:07.781071 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.781043 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:16:07.815281 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.815233 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-tls-certs\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.815281 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.815286 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-model-cache\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.815534 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.815364 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-home\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.815534 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.815398 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-dshm\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.815534 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.815438 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-tmp-dir\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.815534 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.815488 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2jzb\" (UniqueName: \"kubernetes.io/projected/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-kube-api-access-w2jzb\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.916960 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.916920 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-tmp-dir\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.917144 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.916989 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jzb\" (UniqueName: \"kubernetes.io/projected/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-kube-api-access-w2jzb\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.917144 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.917040 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-tls-certs\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.917144 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.917069 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-model-cache\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.917144 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.917099 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-home\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.917144 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.917124 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-dshm\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.917479 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.917452 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-tmp-dir\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.917631 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.917501 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-model-cache\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.917631 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.917520 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-home\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.919381 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.919359 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-dshm\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.919494 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.919478 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-tls-certs\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:07.927161 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:07.927137 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2jzb\" (UniqueName: \"kubernetes.io/projected/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-kube-api-access-w2jzb\") pod \"rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:08.029891 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:08.029858 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:08.168436 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:08.168410 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t"] Jul 22 01:16:08.170584 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:16:08.170558 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb1c3c76_a8f0_49cc_84bb_e98696f5c962.slice/crio-8f4fe3eb970b64ed38e5947166a20adb5dc6a8d77f6e4dabe2c8890a40753ebc WatchSource:0}: Error finding container 8f4fe3eb970b64ed38e5947166a20adb5dc6a8d77f6e4dabe2c8890a40753ebc: Status 404 returned error can't find the container with id 8f4fe3eb970b64ed38e5947166a20adb5dc6a8d77f6e4dabe2c8890a40753ebc Jul 22 01:16:08.233383 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:08.233358 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" event={"ID":"fb1c3c76-a8f0-49cc-84bb-e98696f5c962","Type":"ContainerStarted","Data":"04e9913ab8dc5de0da42eb8dba52ea7cd43ce141ed791dc7d23e1aff60ed39a9"} Jul 22 01:16:08.233495 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:08.233396 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" event={"ID":"fb1c3c76-a8f0-49cc-84bb-e98696f5c962","Type":"ContainerStarted","Data":"8f4fe3eb970b64ed38e5947166a20adb5dc6a8d77f6e4dabe2c8890a40753ebc"} Jul 22 01:16:14.829152 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:14.829119 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:16:14.930054 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:14.930012 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:16:18.030827 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:18.030770 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:18.031235 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:18.030845 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:18.043546 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:18.043522 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:18.061173 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:18.061124 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" podStartSLOduration=11.06111081 podStartE2EDuration="11.06111081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:16:08.25630324 +0000 UTC m=+2976.582375443" watchObservedRunningTime="2026-07-22 01:16:18.06111081 +0000 UTC m=+2986.387183012" Jul 22 01:16:18.294106 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:18.294024 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:16:40.316457 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:40.316331 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 01:16:40.327359 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:40.327335 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g5jvn_07ca19d0-5de0-4391-b15e-2a229d853303/ovn-acl-logging/0.log" Jul 22 01:16:40.638822 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:40.638715 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:16:40.736736 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:40.736705 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:16:50.362964 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:50.362926 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:16:53.863446 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:53.863408 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:16:54.733450 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:54.733414 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t"] Jul 22 01:16:54.758032 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:54.757993 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:16:55.671046 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:55.671011 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7602-140353403390128"] Jul 22 01:16:55.674402 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:55.674383 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353403390128" Jul 22 01:16:55.677363 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:55.677341 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wnzj6\"" Jul 22 01:16:55.679696 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:55.679670 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-140353403390128"] Jul 22 01:16:55.759390 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:55.759357 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz5s6\" (UniqueName: \"kubernetes.io/projected/1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682-kube-api-access-pz5s6\") pod \"test-curl-7602-140353403390128\" (UID: \"1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682\") " pod="opendatahub/test-curl-7602-140353403390128" Jul 22 01:16:55.860012 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:55.859967 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz5s6\" (UniqueName: \"kubernetes.io/projected/1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682-kube-api-access-pz5s6\") pod \"test-curl-7602-140353403390128\" (UID: \"1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682\") " pod="opendatahub/test-curl-7602-140353403390128" Jul 22 01:16:55.868270 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:55.868241 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz5s6\" (UniqueName: \"kubernetes.io/projected/1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682-kube-api-access-pz5s6\") pod \"test-curl-7602-140353403390128\" (UID: \"1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682\") " pod="opendatahub/test-curl-7602-140353403390128" Jul 22 01:16:55.984472 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:55.984382 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353403390128" Jul 22 01:16:56.124435 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:56.124408 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-140353403390128"] Jul 22 01:16:56.126313 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:16:56.126278 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ec15e2a_c01f_4f6b_9ca6_edd8f93cf682.slice/crio-5a18921be6121d65af25493690d0739e0839abcb0d385733f2f37efed4f4e0be WatchSource:0}: Error finding container 5a18921be6121d65af25493690d0739e0839abcb0d385733f2f37efed4f4e0be: Status 404 returned error can't find the container with id 5a18921be6121d65af25493690d0739e0839abcb0d385733f2f37efed4f4e0be Jul 22 01:16:56.127860 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:56.127764 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 01:16:56.479361 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:56.479322 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-140353403390128" event={"ID":"1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682","Type":"ContainerStarted","Data":"5a18921be6121d65af25493690d0739e0839abcb0d385733f2f37efed4f4e0be"} Jul 22 01:16:57.485237 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:57.485202 2567 generic.go:356] "Generic (PLEG): container finished" podID="1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682" containerID="206a2a64afe20b740c3067a8db2c846403d8221101985c0d6a4d03ec9e2be1d9" exitCode=0 Jul 22 01:16:57.485617 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:57.485286 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-140353403390128" event={"ID":"1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682","Type":"ContainerDied","Data":"206a2a64afe20b740c3067a8db2c846403d8221101985c0d6a4d03ec9e2be1d9"} Jul 22 01:16:58.630587 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:58.630563 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353403390128" Jul 22 01:16:58.655356 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:58.655329 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7602-140353403390128_1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682/test-curl-7602-140353403390128/0.log" Jul 22 01:16:58.681830 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:58.681806 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7602-140353403390128"] Jul 22 01:16:58.685817 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:58.685794 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7602-140353403390128"] Jul 22 01:16:58.689280 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:58.689259 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz5s6\" (UniqueName: \"kubernetes.io/projected/1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682-kube-api-access-pz5s6\") pod \"1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682\" (UID: \"1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682\") " Jul 22 01:16:58.691417 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:58.691391 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682-kube-api-access-pz5s6" (OuterVolumeSpecName: "kube-api-access-pz5s6") pod "1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682" (UID: "1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682"). InnerVolumeSpecName "kube-api-access-pz5s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:16:58.790224 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:58.790189 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz5s6\" (UniqueName: \"kubernetes.io/projected/1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682-kube-api-access-pz5s6\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:16:58.805605 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:58.805583 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7602-140353408500400"] Jul 22 01:16:58.805963 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:58.805947 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682" containerName="test-curl-7602-140353403390128" Jul 22 01:16:58.806011 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:58.805964 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682" containerName="test-curl-7602-140353403390128" Jul 22 01:16:58.806047 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:58.806027 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682" containerName="test-curl-7602-140353403390128" Jul 22 01:16:58.809014 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:58.808999 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408500400" Jul 22 01:16:58.818804 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:58.815167 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-140353408500400"] Jul 22 01:16:58.891426 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:58.891398 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgt8h\" (UniqueName: \"kubernetes.io/projected/10d1f3b6-7eb7-43f2-b085-d24f623565d0-kube-api-access-hgt8h\") pod \"test-curl-7602-140353408500400\" (UID: \"10d1f3b6-7eb7-43f2-b085-d24f623565d0\") " pod="opendatahub/test-curl-7602-140353408500400" Jul 22 01:16:58.992669 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:58.992643 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgt8h\" (UniqueName: \"kubernetes.io/projected/10d1f3b6-7eb7-43f2-b085-d24f623565d0-kube-api-access-hgt8h\") pod \"test-curl-7602-140353408500400\" (UID: \"10d1f3b6-7eb7-43f2-b085-d24f623565d0\") " pod="opendatahub/test-curl-7602-140353408500400" Jul 22 01:16:59.000992 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:59.000964 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgt8h\" (UniqueName: \"kubernetes.io/projected/10d1f3b6-7eb7-43f2-b085-d24f623565d0-kube-api-access-hgt8h\") pod \"test-curl-7602-140353408500400\" (UID: \"10d1f3b6-7eb7-43f2-b085-d24f623565d0\") " pod="opendatahub/test-curl-7602-140353408500400" Jul 22 01:16:59.123976 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:59.123906 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408500400" Jul 22 01:16:59.254562 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:59.254539 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-140353408500400"] Jul 22 01:16:59.256818 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:16:59.256788 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10d1f3b6_7eb7_43f2_b085_d24f623565d0.slice/crio-e9fb64aaf5101b98ab231357da6974f46669fdcedcf0c8a50f14bbd3dbedb0b2 WatchSource:0}: Error finding container e9fb64aaf5101b98ab231357da6974f46669fdcedcf0c8a50f14bbd3dbedb0b2: Status 404 returned error can't find the container with id e9fb64aaf5101b98ab231357da6974f46669fdcedcf0c8a50f14bbd3dbedb0b2 Jul 22 01:16:59.496258 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:59.496168 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-140353408500400" event={"ID":"10d1f3b6-7eb7-43f2-b085-d24f623565d0","Type":"ContainerStarted","Data":"e9fb64aaf5101b98ab231357da6974f46669fdcedcf0c8a50f14bbd3dbedb0b2"} Jul 22 01:16:59.497298 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:59.497278 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a18921be6121d65af25493690d0739e0839abcb0d385733f2f37efed4f4e0be" Jul 22 01:16:59.497396 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:16:59.497309 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353403390128" Jul 22 01:16:59.507819 ip-10-0-132-178 kubenswrapper[2567]: E0722 01:16:59.507763 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7602-140353403390128\" is forbidden: User \"system:node:ip-10-0-132-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-178.ec2.internal' and this object" podUID="1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682" pod="opendatahub/test-curl-7602-140353403390128" Jul 22 01:17:00.118967 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:00.118939 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682" path="/var/lib/kubelet/pods/1ec15e2a-c01f-4f6b-9ca6-edd8f93cf682/volumes" Jul 22 01:17:00.503223 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:00.503137 2567 generic.go:356] "Generic (PLEG): container finished" podID="10d1f3b6-7eb7-43f2-b085-d24f623565d0" containerID="33571480432d8a9047e913be063525e2df8182523cafc1ccd704599e55bd65ad" exitCode=0 Jul 22 01:17:00.503367 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:00.503214 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-140353408500400" event={"ID":"10d1f3b6-7eb7-43f2-b085-d24f623565d0","Type":"ContainerDied","Data":"33571480432d8a9047e913be063525e2df8182523cafc1ccd704599e55bd65ad"} Jul 22 01:17:01.651198 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:01.651169 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408500400" Jul 22 01:17:01.673616 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:01.673576 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7602-140353408500400_10d1f3b6-7eb7-43f2-b085-d24f623565d0/test-curl-7602-140353408500400/0.log" Jul 22 01:17:01.696680 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:01.696641 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7602-140353408500400"] Jul 22 01:17:01.699057 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:01.699030 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7602-140353408500400"] Jul 22 01:17:01.718042 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:01.718017 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgt8h\" (UniqueName: \"kubernetes.io/projected/10d1f3b6-7eb7-43f2-b085-d24f623565d0-kube-api-access-hgt8h\") pod \"10d1f3b6-7eb7-43f2-b085-d24f623565d0\" (UID: \"10d1f3b6-7eb7-43f2-b085-d24f623565d0\") " Jul 22 01:17:01.720204 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:01.720177 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d1f3b6-7eb7-43f2-b085-d24f623565d0-kube-api-access-hgt8h" (OuterVolumeSpecName: "kube-api-access-hgt8h") pod "10d1f3b6-7eb7-43f2-b085-d24f623565d0" (UID: "10d1f3b6-7eb7-43f2-b085-d24f623565d0"). InnerVolumeSpecName "kube-api-access-hgt8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:17:01.818810 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:01.818744 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgt8h\" (UniqueName: \"kubernetes.io/projected/10d1f3b6-7eb7-43f2-b085-d24f623565d0-kube-api-access-hgt8h\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:17:01.860360 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:01.860321 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7602-140353408593328"] Jul 22 01:17:01.860750 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:01.860736 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10d1f3b6-7eb7-43f2-b085-d24f623565d0" containerName="test-curl-7602-140353408500400" Jul 22 01:17:01.860844 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:01.860755 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="10d1f3b6-7eb7-43f2-b085-d24f623565d0" containerName="test-curl-7602-140353408500400" Jul 22 01:17:01.860891 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:01.860862 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="10d1f3b6-7eb7-43f2-b085-d24f623565d0" containerName="test-curl-7602-140353408500400" Jul 22 01:17:01.863946 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:01.863930 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408593328" Jul 22 01:17:01.867953 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:01.867926 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-140353408593328"] Jul 22 01:17:01.919868 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:01.919764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvbbh\" (UniqueName: \"kubernetes.io/projected/2591c12a-305c-424d-b1b6-bb7b1c533d57-kube-api-access-rvbbh\") pod \"test-curl-7602-140353408593328\" (UID: \"2591c12a-305c-424d-b1b6-bb7b1c533d57\") " pod="opendatahub/test-curl-7602-140353408593328" Jul 22 01:17:02.020923 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:02.020888 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvbbh\" (UniqueName: \"kubernetes.io/projected/2591c12a-305c-424d-b1b6-bb7b1c533d57-kube-api-access-rvbbh\") pod \"test-curl-7602-140353408593328\" (UID: \"2591c12a-305c-424d-b1b6-bb7b1c533d57\") " pod="opendatahub/test-curl-7602-140353408593328" Jul 22 01:17:02.029078 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:02.029056 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvbbh\" (UniqueName: \"kubernetes.io/projected/2591c12a-305c-424d-b1b6-bb7b1c533d57-kube-api-access-rvbbh\") pod \"test-curl-7602-140353408593328\" (UID: \"2591c12a-305c-424d-b1b6-bb7b1c533d57\") " pod="opendatahub/test-curl-7602-140353408593328" Jul 22 01:17:02.118983 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:02.118943 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10d1f3b6-7eb7-43f2-b085-d24f623565d0" path="/var/lib/kubelet/pods/10d1f3b6-7eb7-43f2-b085-d24f623565d0/volumes" Jul 22 01:17:02.174471 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:02.174386 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408593328" Jul 22 01:17:02.515099 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:02.515075 2567 scope.go:117] "RemoveContainer" containerID="33571480432d8a9047e913be063525e2df8182523cafc1ccd704599e55bd65ad" Jul 22 01:17:02.515099 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:02.515091 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408500400" Jul 22 01:17:02.523196 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:02.523174 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-140353408593328"] Jul 22 01:17:02.525936 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:17:02.525910 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2591c12a_305c_424d_b1b6_bb7b1c533d57.slice/crio-0f66c7ba109acb76bc949c3c9e2092e4009efb0753123c8ff8b29b35b67b157c WatchSource:0}: Error finding container 0f66c7ba109acb76bc949c3c9e2092e4009efb0753123c8ff8b29b35b67b157c: Status 404 returned error can't find the container with id 0f66c7ba109acb76bc949c3c9e2092e4009efb0753123c8ff8b29b35b67b157c Jul 22 01:17:03.521386 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:03.521348 2567 generic.go:356] "Generic (PLEG): container finished" podID="2591c12a-305c-424d-b1b6-bb7b1c533d57" containerID="937c0b3304869ec7c08500f4da29fcca27c98b2454275523c2691b5584bfe9ae" exitCode=0 Jul 22 01:17:03.521817 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:03.521408 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-140353408593328" event={"ID":"2591c12a-305c-424d-b1b6-bb7b1c533d57","Type":"ContainerDied","Data":"937c0b3304869ec7c08500f4da29fcca27c98b2454275523c2691b5584bfe9ae"} Jul 22 01:17:03.521817 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:03.521455 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-140353408593328" event={"ID":"2591c12a-305c-424d-b1b6-bb7b1c533d57","Type":"ContainerStarted","Data":"0f66c7ba109acb76bc949c3c9e2092e4009efb0753123c8ff8b29b35b67b157c"} Jul 22 01:17:04.662902 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:04.662873 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408593328" Jul 22 01:17:04.689637 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:04.689614 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7602-140353408593328_2591c12a-305c-424d-b1b6-bb7b1c533d57/test-curl-7602-140353408593328/0.log" Jul 22 01:17:04.714870 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:04.714842 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7602-140353408593328"] Jul 22 01:17:04.717051 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:04.717024 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7602-140353408593328"] Jul 22 01:17:04.745155 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:04.745131 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvbbh\" (UniqueName: \"kubernetes.io/projected/2591c12a-305c-424d-b1b6-bb7b1c533d57-kube-api-access-rvbbh\") pod \"2591c12a-305c-424d-b1b6-bb7b1c533d57\" (UID: \"2591c12a-305c-424d-b1b6-bb7b1c533d57\") " Jul 22 01:17:04.747112 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:04.747091 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2591c12a-305c-424d-b1b6-bb7b1c533d57-kube-api-access-rvbbh" (OuterVolumeSpecName: "kube-api-access-rvbbh") pod "2591c12a-305c-424d-b1b6-bb7b1c533d57" (UID: "2591c12a-305c-424d-b1b6-bb7b1c533d57"). InnerVolumeSpecName "kube-api-access-rvbbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:17:04.845727 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:04.845653 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7602-140353408593328"] Jul 22 01:17:04.846171 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:04.846146 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvbbh\" (UniqueName: \"kubernetes.io/projected/2591c12a-305c-424d-b1b6-bb7b1c533d57-kube-api-access-rvbbh\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:17:04.846238 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:04.846224 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2591c12a-305c-424d-b1b6-bb7b1c533d57" containerName="test-curl-7602-140353408593328" Jul 22 01:17:04.846272 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:04.846241 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2591c12a-305c-424d-b1b6-bb7b1c533d57" containerName="test-curl-7602-140353408593328" Jul 22 01:17:04.846328 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:04.846319 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2591c12a-305c-424d-b1b6-bb7b1c533d57" containerName="test-curl-7602-140353408593328" Jul 22 01:17:04.849345 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:04.849329 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408593328" Jul 22 01:17:04.855806 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:04.855764 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-140353408593328"] Jul 22 01:17:04.946866 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:04.946838 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzc42\" (UniqueName: \"kubernetes.io/projected/eff75593-ee97-40f5-8ef9-3372bb9f564d-kube-api-access-wzc42\") pod \"test-curl-7602-140353408593328\" (UID: \"eff75593-ee97-40f5-8ef9-3372bb9f564d\") " pod="opendatahub/test-curl-7602-140353408593328" Jul 22 01:17:05.047401 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:05.047356 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzc42\" (UniqueName: \"kubernetes.io/projected/eff75593-ee97-40f5-8ef9-3372bb9f564d-kube-api-access-wzc42\") pod \"test-curl-7602-140353408593328\" (UID: \"eff75593-ee97-40f5-8ef9-3372bb9f564d\") " pod="opendatahub/test-curl-7602-140353408593328" Jul 22 01:17:05.055851 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:05.055819 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzc42\" (UniqueName: \"kubernetes.io/projected/eff75593-ee97-40f5-8ef9-3372bb9f564d-kube-api-access-wzc42\") pod \"test-curl-7602-140353408593328\" (UID: \"eff75593-ee97-40f5-8ef9-3372bb9f564d\") " pod="opendatahub/test-curl-7602-140353408593328" Jul 22 01:17:05.159275 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:05.159171 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408593328" Jul 22 01:17:05.293419 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:05.293391 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-140353408593328"] Jul 22 01:17:05.294654 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:17:05.294626 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeff75593_ee97_40f5_8ef9_3372bb9f564d.slice/crio-fb3f8a566498daae99f0d09e625d9cf923ce56bef4a15d3403c4f2efc2f0377f WatchSource:0}: Error finding container fb3f8a566498daae99f0d09e625d9cf923ce56bef4a15d3403c4f2efc2f0377f: Status 404 returned error can't find the container with id fb3f8a566498daae99f0d09e625d9cf923ce56bef4a15d3403c4f2efc2f0377f Jul 22 01:17:05.540051 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:05.540012 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408593328" Jul 22 01:17:05.540253 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:05.540017 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f66c7ba109acb76bc949c3c9e2092e4009efb0753123c8ff8b29b35b67b157c" Jul 22 01:17:05.541191 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:05.541170 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-140353408593328" event={"ID":"eff75593-ee97-40f5-8ef9-3372bb9f564d","Type":"ContainerStarted","Data":"fb3f8a566498daae99f0d09e625d9cf923ce56bef4a15d3403c4f2efc2f0377f"} Jul 22 01:17:05.551830 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:05.551774 2567 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7602-140353408593328" oldPodUID="2591c12a-305c-424d-b1b6-bb7b1c533d57" podUID="eff75593-ee97-40f5-8ef9-3372bb9f564d" Jul 22 01:17:06.120512 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:06.120481 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2591c12a-305c-424d-b1b6-bb7b1c533d57" path="/var/lib/kubelet/pods/2591c12a-305c-424d-b1b6-bb7b1c533d57/volumes" Jul 22 01:17:06.547910 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:06.547870 2567 generic.go:356] "Generic (PLEG): container finished" podID="eff75593-ee97-40f5-8ef9-3372bb9f564d" containerID="61be59d5abbe6c790a40775451398ff53c7a9a81bc22250208862ad317bc0a6a" exitCode=0 Jul 22 01:17:06.548100 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:06.547955 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-140353408593328" event={"ID":"eff75593-ee97-40f5-8ef9-3372bb9f564d","Type":"ContainerDied","Data":"61be59d5abbe6c790a40775451398ff53c7a9a81bc22250208862ad317bc0a6a"} Jul 22 01:17:07.687462 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:07.687433 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408593328" Jul 22 01:17:07.708186 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:07.708158 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7602-140353408593328_eff75593-ee97-40f5-8ef9-3372bb9f564d/test-curl-7602-140353408593328/0.log" Jul 22 01:17:07.733735 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:07.733702 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7602-140353408593328"] Jul 22 01:17:07.738981 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:07.738948 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7602-140353408593328"] Jul 22 01:17:07.772385 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:07.772352 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzc42\" (UniqueName: \"kubernetes.io/projected/eff75593-ee97-40f5-8ef9-3372bb9f564d-kube-api-access-wzc42\") pod \"eff75593-ee97-40f5-8ef9-3372bb9f564d\" (UID: \"eff75593-ee97-40f5-8ef9-3372bb9f564d\") " Jul 22 01:17:07.774393 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:07.774357 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff75593-ee97-40f5-8ef9-3372bb9f564d-kube-api-access-wzc42" (OuterVolumeSpecName: "kube-api-access-wzc42") pod "eff75593-ee97-40f5-8ef9-3372bb9f564d" (UID: "eff75593-ee97-40f5-8ef9-3372bb9f564d"). InnerVolumeSpecName "kube-api-access-wzc42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:17:07.866261 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:07.866178 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7602-140353408459232"] Jul 22 01:17:07.866577 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:07.866564 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff75593-ee97-40f5-8ef9-3372bb9f564d" containerName="test-curl-7602-140353408593328" Jul 22 01:17:07.866616 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:07.866579 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff75593-ee97-40f5-8ef9-3372bb9f564d" containerName="test-curl-7602-140353408593328" Jul 22 01:17:07.866667 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:07.866658 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff75593-ee97-40f5-8ef9-3372bb9f564d" containerName="test-curl-7602-140353408593328" Jul 22 01:17:07.869607 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:07.869590 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408459232" Jul 22 01:17:07.873107 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:07.873079 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzc42\" (UniqueName: \"kubernetes.io/projected/eff75593-ee97-40f5-8ef9-3372bb9f564d-kube-api-access-wzc42\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:17:07.874181 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:07.874157 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-140353408459232"] Jul 22 01:17:07.974678 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:07.974642 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj6jr\" (UniqueName: \"kubernetes.io/projected/ddb570aa-a671-469b-bb0c-cf15b9b2930c-kube-api-access-xj6jr\") pod \"test-curl-7602-140353408459232\" (UID: \"ddb570aa-a671-469b-bb0c-cf15b9b2930c\") " pod="opendatahub/test-curl-7602-140353408459232" Jul 22 01:17:08.075761 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:08.075719 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj6jr\" (UniqueName: \"kubernetes.io/projected/ddb570aa-a671-469b-bb0c-cf15b9b2930c-kube-api-access-xj6jr\") pod \"test-curl-7602-140353408459232\" (UID: \"ddb570aa-a671-469b-bb0c-cf15b9b2930c\") " pod="opendatahub/test-curl-7602-140353408459232" Jul 22 01:17:08.084625 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:08.084602 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj6jr\" (UniqueName: \"kubernetes.io/projected/ddb570aa-a671-469b-bb0c-cf15b9b2930c-kube-api-access-xj6jr\") pod \"test-curl-7602-140353408459232\" (UID: \"ddb570aa-a671-469b-bb0c-cf15b9b2930c\") " pod="opendatahub/test-curl-7602-140353408459232" Jul 22 01:17:08.118708 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:08.118630 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff75593-ee97-40f5-8ef9-3372bb9f564d" path="/var/lib/kubelet/pods/eff75593-ee97-40f5-8ef9-3372bb9f564d/volumes" Jul 22 01:17:08.180195 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:08.180157 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408459232" Jul 22 01:17:08.316865 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:08.316839 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-140353408459232"] Jul 22 01:17:08.318708 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:17:08.318672 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddb570aa_a671_469b_bb0c_cf15b9b2930c.slice/crio-10ace9e91bcc63716ed13e1a3365cb11a7f6b3fb1b9f2042509eba123b9a97d1 WatchSource:0}: Error finding container 10ace9e91bcc63716ed13e1a3365cb11a7f6b3fb1b9f2042509eba123b9a97d1: Status 404 returned error can't find the container with id 10ace9e91bcc63716ed13e1a3365cb11a7f6b3fb1b9f2042509eba123b9a97d1 Jul 22 01:17:08.560053 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:08.560018 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-140353408459232" event={"ID":"ddb570aa-a671-469b-bb0c-cf15b9b2930c","Type":"ContainerStarted","Data":"10ace9e91bcc63716ed13e1a3365cb11a7f6b3fb1b9f2042509eba123b9a97d1"} Jul 22 01:17:08.561178 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:08.561159 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408593328" Jul 22 01:17:08.561276 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:08.561161 2567 scope.go:117] "RemoveContainer" containerID="61be59d5abbe6c790a40775451398ff53c7a9a81bc22250208862ad317bc0a6a" Jul 22 01:17:09.567812 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:09.567760 2567 generic.go:356] "Generic (PLEG): container finished" podID="ddb570aa-a671-469b-bb0c-cf15b9b2930c" containerID="f8c2116f62c040e7f375086390c3718d8d948ec9735b5f29f2c468380e9801b0" exitCode=0 Jul 22 01:17:09.568271 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:09.567850 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-140353408459232" event={"ID":"ddb570aa-a671-469b-bb0c-cf15b9b2930c","Type":"ContainerDied","Data":"f8c2116f62c040e7f375086390c3718d8d948ec9735b5f29f2c468380e9801b0"} Jul 22 01:17:09.753465 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:09.753420 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" podUID="fb1c3c76-a8f0-49cc-84bb-e98696f5c962" containerName="main" containerID="cri-o://04e9913ab8dc5de0da42eb8dba52ea7cd43ce141ed791dc7d23e1aff60ed39a9" gracePeriod=45 Jul 22 01:17:10.011416 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.011389 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:17:10.096324 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.096283 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-tmp-dir\") pod \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " Jul 22 01:17:10.096324 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.096332 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-model-cache\") pod \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " Jul 22 01:17:10.096589 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.096392 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2jzb\" (UniqueName: \"kubernetes.io/projected/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-kube-api-access-w2jzb\") pod \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " Jul 22 01:17:10.096589 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.096440 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-home\") pod \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " Jul 22 01:17:10.096589 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.096473 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-dshm\") pod \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " Jul 22 01:17:10.096589 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.096512 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-tls-certs\") pod \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\" (UID: \"fb1c3c76-a8f0-49cc-84bb-e98696f5c962\") " Jul 22 01:17:10.096811 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.096590 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb1c3c76-a8f0-49cc-84bb-e98696f5c962" (UID: "fb1c3c76-a8f0-49cc-84bb-e98696f5c962"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:17:10.096811 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.096640 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb1c3c76-a8f0-49cc-84bb-e98696f5c962" (UID: "fb1c3c76-a8f0-49cc-84bb-e98696f5c962"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:17:10.096811 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.096752 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-home" (OuterVolumeSpecName: "home") pod "fb1c3c76-a8f0-49cc-84bb-e98696f5c962" (UID: "fb1c3c76-a8f0-49cc-84bb-e98696f5c962"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:17:10.096960 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.096911 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-tmp-dir\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:17:10.096960 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.096929 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-model-cache\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:17:10.096960 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.096937 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-home\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:17:10.098645 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.098619 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-dshm" (OuterVolumeSpecName: "dshm") pod "fb1c3c76-a8f0-49cc-84bb-e98696f5c962" (UID: "fb1c3c76-a8f0-49cc-84bb-e98696f5c962"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:17:10.098718 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.098693 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb1c3c76-a8f0-49cc-84bb-e98696f5c962" (UID: "fb1c3c76-a8f0-49cc-84bb-e98696f5c962"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:17:10.098756 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.098693 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-kube-api-access-w2jzb" (OuterVolumeSpecName: "kube-api-access-w2jzb") pod "fb1c3c76-a8f0-49cc-84bb-e98696f5c962" (UID: "fb1c3c76-a8f0-49cc-84bb-e98696f5c962"). InnerVolumeSpecName "kube-api-access-w2jzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:17:10.198336 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.198294 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2jzb\" (UniqueName: \"kubernetes.io/projected/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-kube-api-access-w2jzb\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:17:10.198336 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.198329 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-dshm\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:17:10.198336 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.198339 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1c3c76-a8f0-49cc-84bb-e98696f5c962-tls-certs\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:17:10.574592 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.574560 2567 generic.go:356] "Generic (PLEG): container finished" podID="fb1c3c76-a8f0-49cc-84bb-e98696f5c962" containerID="04e9913ab8dc5de0da42eb8dba52ea7cd43ce141ed791dc7d23e1aff60ed39a9" exitCode=0 Jul 22 01:17:10.575066 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.574601 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" Jul 22 01:17:10.575066 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.574647 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" event={"ID":"fb1c3c76-a8f0-49cc-84bb-e98696f5c962","Type":"ContainerDied","Data":"04e9913ab8dc5de0da42eb8dba52ea7cd43ce141ed791dc7d23e1aff60ed39a9"} Jul 22 01:17:10.575066 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.574684 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t" event={"ID":"fb1c3c76-a8f0-49cc-84bb-e98696f5c962","Type":"ContainerDied","Data":"8f4fe3eb970b64ed38e5947166a20adb5dc6a8d77f6e4dabe2c8890a40753ebc"} Jul 22 01:17:10.575066 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.574707 2567 scope.go:117] "RemoveContainer" containerID="04e9913ab8dc5de0da42eb8dba52ea7cd43ce141ed791dc7d23e1aff60ed39a9" Jul 22 01:17:10.583620 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.583599 2567 scope.go:117] "RemoveContainer" containerID="04e9913ab8dc5de0da42eb8dba52ea7cd43ce141ed791dc7d23e1aff60ed39a9" Jul 22 01:17:10.583962 ip-10-0-132-178 kubenswrapper[2567]: E0722 01:17:10.583940 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e9913ab8dc5de0da42eb8dba52ea7cd43ce141ed791dc7d23e1aff60ed39a9\": container with ID starting with 04e9913ab8dc5de0da42eb8dba52ea7cd43ce141ed791dc7d23e1aff60ed39a9 not found: ID does not exist" containerID="04e9913ab8dc5de0da42eb8dba52ea7cd43ce141ed791dc7d23e1aff60ed39a9" Jul 22 01:17:10.584053 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.583972 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e9913ab8dc5de0da42eb8dba52ea7cd43ce141ed791dc7d23e1aff60ed39a9"} err="failed to get container status \"04e9913ab8dc5de0da42eb8dba52ea7cd43ce141ed791dc7d23e1aff60ed39a9\": rpc error: code = NotFound desc = could not find container \"04e9913ab8dc5de0da42eb8dba52ea7cd43ce141ed791dc7d23e1aff60ed39a9\": container with ID starting with 04e9913ab8dc5de0da42eb8dba52ea7cd43ce141ed791dc7d23e1aff60ed39a9 not found: ID does not exist" Jul 22 01:17:10.595269 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.595227 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t"] Jul 22 01:17:10.598520 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.598491 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4f35c1/rate-test-model-4f35c1-kserve-8c4977fbc-9nq5t"] Jul 22 01:17:10.700210 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.700184 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408459232" Jul 22 01:17:10.721242 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.721218 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7602-140353408459232_ddb570aa-a671-469b-bb0c-cf15b9b2930c/test-curl-7602-140353408459232/0.log" Jul 22 01:17:10.744993 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.744957 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7602-140353408459232"] Jul 22 01:17:10.748496 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.748470 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7602-140353408459232"] Jul 22 01:17:10.804151 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.804115 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj6jr\" (UniqueName: \"kubernetes.io/projected/ddb570aa-a671-469b-bb0c-cf15b9b2930c-kube-api-access-xj6jr\") pod \"ddb570aa-a671-469b-bb0c-cf15b9b2930c\" (UID: \"ddb570aa-a671-469b-bb0c-cf15b9b2930c\") " Jul 22 01:17:10.806144 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.806113 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb570aa-a671-469b-bb0c-cf15b9b2930c-kube-api-access-xj6jr" (OuterVolumeSpecName: "kube-api-access-xj6jr") pod "ddb570aa-a671-469b-bb0c-cf15b9b2930c" (UID: "ddb570aa-a671-469b-bb0c-cf15b9b2930c"). InnerVolumeSpecName "kube-api-access-xj6jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:17:10.874234 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.874165 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7602-140353408459232"] Jul 22 01:17:10.874549 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.874535 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1c3c76-a8f0-49cc-84bb-e98696f5c962" containerName="main" Jul 22 01:17:10.874549 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.874550 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1c3c76-a8f0-49cc-84bb-e98696f5c962" containerName="main" Jul 22 01:17:10.874658 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.874560 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddb570aa-a671-469b-bb0c-cf15b9b2930c" containerName="test-curl-7602-140353408459232" Jul 22 01:17:10.874658 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.874567 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddb570aa-a671-469b-bb0c-cf15b9b2930c" containerName="test-curl-7602-140353408459232" Jul 22 01:17:10.874658 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.874651 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddb570aa-a671-469b-bb0c-cf15b9b2930c" containerName="test-curl-7602-140353408459232" Jul 22 01:17:10.874757 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.874661 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb1c3c76-a8f0-49cc-84bb-e98696f5c962" containerName="main" Jul 22 01:17:10.877726 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.877709 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408459232" Jul 22 01:17:10.884485 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.884461 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-140353408459232"] Jul 22 01:17:10.905724 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:10.905697 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj6jr\" (UniqueName: \"kubernetes.io/projected/ddb570aa-a671-469b-bb0c-cf15b9b2930c-kube-api-access-xj6jr\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:17:11.006265 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:11.006229 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59k7m\" (UniqueName: \"kubernetes.io/projected/c11dbbc0-e431-4e81-8645-376768e28fb8-kube-api-access-59k7m\") pod \"test-curl-7602-140353408459232\" (UID: \"c11dbbc0-e431-4e81-8645-376768e28fb8\") " pod="opendatahub/test-curl-7602-140353408459232" Jul 22 01:17:11.107212 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:11.107177 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59k7m\" (UniqueName: \"kubernetes.io/projected/c11dbbc0-e431-4e81-8645-376768e28fb8-kube-api-access-59k7m\") pod \"test-curl-7602-140353408459232\" (UID: \"c11dbbc0-e431-4e81-8645-376768e28fb8\") " pod="opendatahub/test-curl-7602-140353408459232" Jul 22 01:17:11.116110 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:11.116087 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59k7m\" (UniqueName: \"kubernetes.io/projected/c11dbbc0-e431-4e81-8645-376768e28fb8-kube-api-access-59k7m\") pod \"test-curl-7602-140353408459232\" (UID: \"c11dbbc0-e431-4e81-8645-376768e28fb8\") " pod="opendatahub/test-curl-7602-140353408459232" Jul 22 01:17:11.187553 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:11.187482 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408459232" Jul 22 01:17:11.315213 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:11.315040 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-140353408459232"] Jul 22 01:17:11.320408 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:17:11.320368 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc11dbbc0_e431_4e81_8645_376768e28fb8.slice/crio-e12970c2b5a170f72aa7a67f78e8f0f6c4863d7bc802824fbd471c7446c6d53d WatchSource:0}: Error finding container e12970c2b5a170f72aa7a67f78e8f0f6c4863d7bc802824fbd471c7446c6d53d: Status 404 returned error can't find the container with id e12970c2b5a170f72aa7a67f78e8f0f6c4863d7bc802824fbd471c7446c6d53d Jul 22 01:17:11.582593 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:11.582557 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-140353408459232" event={"ID":"c11dbbc0-e431-4e81-8645-376768e28fb8","Type":"ContainerStarted","Data":"e12970c2b5a170f72aa7a67f78e8f0f6c4863d7bc802824fbd471c7446c6d53d"} Jul 22 01:17:11.583799 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:11.583759 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10ace9e91bcc63716ed13e1a3365cb11a7f6b3fb1b9f2042509eba123b9a97d1" Jul 22 01:17:11.583928 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:11.583804 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408459232" Jul 22 01:17:11.595521 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:11.595496 2567 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7602-140353408459232" oldPodUID="ddb570aa-a671-469b-bb0c-cf15b9b2930c" podUID="c11dbbc0-e431-4e81-8645-376768e28fb8" Jul 22 01:17:12.120101 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:12.120071 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddb570aa-a671-469b-bb0c-cf15b9b2930c" path="/var/lib/kubelet/pods/ddb570aa-a671-469b-bb0c-cf15b9b2930c/volumes" Jul 22 01:17:12.120481 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:12.120468 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb1c3c76-a8f0-49cc-84bb-e98696f5c962" path="/var/lib/kubelet/pods/fb1c3c76-a8f0-49cc-84bb-e98696f5c962/volumes" Jul 22 01:17:12.590332 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:12.590296 2567 generic.go:356] "Generic (PLEG): container finished" podID="c11dbbc0-e431-4e81-8645-376768e28fb8" containerID="cd15b1511b3d1095902e29bd84c42ae268f54dc59fe4a41cfde405211b067fd2" exitCode=0 Jul 22 01:17:12.590725 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:12.590381 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-140353408459232" event={"ID":"c11dbbc0-e431-4e81-8645-376768e28fb8","Type":"ContainerDied","Data":"cd15b1511b3d1095902e29bd84c42ae268f54dc59fe4a41cfde405211b067fd2"} Jul 22 01:17:13.732300 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:13.732270 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408459232" Jul 22 01:17:13.753536 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:13.753505 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7602-140353408459232_c11dbbc0-e431-4e81-8645-376768e28fb8/test-curl-7602-140353408459232/0.log" Jul 22 01:17:13.780172 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:13.780131 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7602-140353408459232"] Jul 22 01:17:13.783767 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:13.783739 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7602-140353408459232"] Jul 22 01:17:13.828251 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:13.828215 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59k7m\" (UniqueName: \"kubernetes.io/projected/c11dbbc0-e431-4e81-8645-376768e28fb8-kube-api-access-59k7m\") pod \"c11dbbc0-e431-4e81-8645-376768e28fb8\" (UID: \"c11dbbc0-e431-4e81-8645-376768e28fb8\") " Jul 22 01:17:13.830394 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:13.830366 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11dbbc0-e431-4e81-8645-376768e28fb8-kube-api-access-59k7m" (OuterVolumeSpecName: "kube-api-access-59k7m") pod "c11dbbc0-e431-4e81-8645-376768e28fb8" (UID: "c11dbbc0-e431-4e81-8645-376768e28fb8"). InnerVolumeSpecName "kube-api-access-59k7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:17:13.929582 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:13.929498 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59k7m\" (UniqueName: \"kubernetes.io/projected/c11dbbc0-e431-4e81-8645-376768e28fb8-kube-api-access-59k7m\") on node \"ip-10-0-132-178.ec2.internal\" DevicePath \"\"" Jul 22 01:17:14.119406 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:14.119364 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c11dbbc0-e431-4e81-8645-376768e28fb8" path="/var/lib/kubelet/pods/c11dbbc0-e431-4e81-8645-376768e28fb8/volumes" Jul 22 01:17:14.602540 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:14.602508 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-140353408459232" Jul 22 01:17:14.602709 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:14.602542 2567 scope.go:117] "RemoveContainer" containerID="cd15b1511b3d1095902e29bd84c42ae268f54dc59fe4a41cfde405211b067fd2" Jul 22 01:17:14.694123 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:14.694071 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:14.797598 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:14.797568 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:24.455814 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:24.455758 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:29.081113 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:29.081021 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:29.463101 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:29.463009 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:32.395807 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:32.395750 2567 scope.go:117] "RemoveContainer" containerID="ad40ec3b30b18369436ddebd26ce5e7f6ec31c3d355d3415d10c24c0bfa01048" Jul 22 01:17:35.761180 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:35.761148 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:35.865000 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:35.864967 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:36.040715 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:36.040622 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:36.277620 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:36.277578 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:37.057427 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:37.057394 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:37.163104 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:37.163068 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:46.343072 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:46.343033 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:46.444020 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:46.443972 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:46.863036 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:46.863005 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:51.461327 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:51.461288 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:51.852931 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:51.852892 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:57.785026 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:57.784980 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:57.886035 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:57.885990 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:58.059530 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:58.059439 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:58.291412 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:58.291375 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:17:58.755501 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:17:58.755470 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-586fc58b47-894bc_5193d573-58ee-4ea7-b823-c8ffc05cd95e/authorino/0.log" Jul 22 01:18:05.208192 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:05.208148 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb84b476f-vg4gh_c5e7758c-a583-43fc-b0d3-06855c41cbbd/manager/0.log" Jul 22 01:18:05.223013 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:05.222983 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb84b476f-vg4gh_c5e7758c-a583-43fc-b0d3-06855c41cbbd/copy-manifests/0.log" Jul 22 01:18:07.393673 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:07.393641 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-586fc58b47-894bc_5193d573-58ee-4ea7-b823-c8ffc05cd95e/authorino/0.log" Jul 22 01:18:07.745656 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:07.745571 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8jlvt_6339b835-255f-4abf-90c3-8a26cee5522f/kuadrant-console-plugin/0.log" Jul 22 01:18:08.913904 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:08.913872 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6467ff7b5c-vxdwq_004b59a0-37e8-4897-9f04-58797d56b611/kube-auth-proxy/0.log" Jul 22 01:18:09.172730 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:09.172646 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xphnd_7fa3fb06-305b-4d41-91cc-ab7a360ef18f/payload-pre-processing/0.log" Jul 22 01:18:09.309179 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:09.309146 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-xx7xj_69b48eb0-702c-4da5-992d-07def03753ca/payload-processing/0.log" Jul 22 01:18:09.425948 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:09.425856 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7584584bbd-c77b7_09532874-5a40-44cd-b95b-1b3e9ad8ecfd/router/0.log" Jul 22 01:18:09.913683 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:09.913653 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-vq6jj_8268b328-5b2f-4c1d-bfd4-83a74c5df038/main/0.log" Jul 22 01:18:10.033811 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:10.033766 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-5hrr8_7e25da29-4761-441b-bb94-77fbc447537b/main/0.log" Jul 22 01:18:10.269659 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:10.269633 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-f4pcs_8effc54f-fa47-4e08-88ee-c49ea54855db/main/0.log" Jul 22 01:18:16.180095 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.180054 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sfz9r/must-gather-qpmls"] Jul 22 01:18:16.180615 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.180592 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c11dbbc0-e431-4e81-8645-376768e28fb8" containerName="test-curl-7602-140353408459232" Jul 22 01:18:16.180615 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.180616 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c11dbbc0-e431-4e81-8645-376768e28fb8" containerName="test-curl-7602-140353408459232" Jul 22 01:18:16.180718 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.180686 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c11dbbc0-e431-4e81-8645-376768e28fb8" containerName="test-curl-7602-140353408459232" Jul 22 01:18:16.183919 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.183900 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sfz9r/must-gather-qpmls" Jul 22 01:18:16.186531 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.186508 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sfz9r\"/\"openshift-service-ca.crt\"" Jul 22 01:18:16.186640 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.186548 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sfz9r\"/\"default-dockercfg-8wxxf\"" Jul 22 01:18:16.187687 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.187673 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sfz9r\"/\"kube-root-ca.crt\"" Jul 22 01:18:16.192560 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.192533 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:18:16.193132 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.193109 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sfz9r/must-gather-qpmls"] Jul 22 01:18:16.200825 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.200803 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9lp9\" (UniqueName: \"kubernetes.io/projected/370207bb-b410-4402-9730-d0e5515a5a4b-kube-api-access-s9lp9\") pod \"must-gather-qpmls\" (UID: \"370207bb-b410-4402-9730-d0e5515a5a4b\") " pod="openshift-must-gather-sfz9r/must-gather-qpmls" Jul 22 01:18:16.200979 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.200943 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/370207bb-b410-4402-9730-d0e5515a5a4b-must-gather-output\") pod \"must-gather-qpmls\" (UID: \"370207bb-b410-4402-9730-d0e5515a5a4b\") " pod="openshift-must-gather-sfz9r/must-gather-qpmls" Jul 22 01:18:16.288950 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.288915 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" limit=1024 Jul 22 01:18:16.302115 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.302089 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9lp9\" (UniqueName: \"kubernetes.io/projected/370207bb-b410-4402-9730-d0e5515a5a4b-kube-api-access-s9lp9\") pod \"must-gather-qpmls\" (UID: \"370207bb-b410-4402-9730-d0e5515a5a4b\") " pod="openshift-must-gather-sfz9r/must-gather-qpmls" Jul 22 01:18:16.302240 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.302155 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/370207bb-b410-4402-9730-d0e5515a5a4b-must-gather-output\") pod \"must-gather-qpmls\" (UID: \"370207bb-b410-4402-9730-d0e5515a5a4b\") " pod="openshift-must-gather-sfz9r/must-gather-qpmls" Jul 22 01:18:16.302457 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.302443 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/370207bb-b410-4402-9730-d0e5515a5a4b-must-gather-output\") pod \"must-gather-qpmls\" (UID: \"370207bb-b410-4402-9730-d0e5515a5a4b\") " pod="openshift-must-gather-sfz9r/must-gather-qpmls" Jul 22 01:18:16.315294 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.315269 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9lp9\" (UniqueName: \"kubernetes.io/projected/370207bb-b410-4402-9730-d0e5515a5a4b-kube-api-access-s9lp9\") pod \"must-gather-qpmls\" (UID: \"370207bb-b410-4402-9730-d0e5515a5a4b\") " pod="openshift-must-gather-sfz9r/must-gather-qpmls" Jul 22 01:18:16.494881 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.494752 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sfz9r/must-gather-qpmls" Jul 22 01:18:16.628837 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.628812 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sfz9r/must-gather-qpmls"] Jul 22 01:18:16.630879 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:18:16.630851 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod370207bb_b410_4402_9730_d0e5515a5a4b.slice/crio-21998290ec3c11e725c6a61cba0686332baef4b52630f0088edd9fee99088182 WatchSource:0}: Error finding container 21998290ec3c11e725c6a61cba0686332baef4b52630f0088edd9fee99088182: Status 404 returned error can't find the container with id 21998290ec3c11e725c6a61cba0686332baef4b52630f0088edd9fee99088182 Jul 22 01:18:16.924587 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:16.924552 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sfz9r/must-gather-qpmls" event={"ID":"370207bb-b410-4402-9730-d0e5515a5a4b","Type":"ContainerStarted","Data":"21998290ec3c11e725c6a61cba0686332baef4b52630f0088edd9fee99088182"} Jul 22 01:18:17.947556 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:17.946098 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sfz9r/must-gather-qpmls" event={"ID":"370207bb-b410-4402-9730-d0e5515a5a4b","Type":"ContainerStarted","Data":"c22ec9106e0472f3212c06fbdb37d9ddaa394511d94bcd844e43dcfbd7a76621"} Jul 22 01:18:17.947556 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:17.946155 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sfz9r/must-gather-qpmls" event={"ID":"370207bb-b410-4402-9730-d0e5515a5a4b","Type":"ContainerStarted","Data":"55588983163bccea67e8c01befbb757f676e40ed5e54acd07a9738fb80c7b832"} Jul 22 01:18:17.968560 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:17.968497 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sfz9r/must-gather-qpmls" podStartSLOduration=1.160268264 podStartE2EDuration="1.968476196s" totalImagesPullingTime="808.207932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:18:17.966696102 +0000 UTC m=+3106.292768313" watchObservedRunningTime="2026-07-22 01:18:17.968476196 +0000 UTC m=+3106.294548399" Jul 22 01:18:19.047790 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:19.047752 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-trczs_30d68a8f-032a-4d0e-90bf-6f70e5b548f2/global-pull-secret-syncer/0.log" Jul 22 01:18:19.124559 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:19.124525 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4wfbw_f1ef826a-11c1-4baf-82fc-134c519b472a/konnectivity-agent/0.log" Jul 22 01:18:19.253821 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:19.253496 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-178.ec2.internal_c36159d4fd5288301a0516429f295afe/haproxy/0.log" Jul 22 01:18:24.998799 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:24.997655 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-586fc58b47-894bc_5193d573-58ee-4ea7-b823-c8ffc05cd95e/authorino/0.log" Jul 22 01:18:25.111034 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:25.110990 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8jlvt_6339b835-255f-4abf-90c3-8a26cee5522f/kuadrant-console-plugin/0.log" Jul 22 01:18:27.002101 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:27.002025 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_80e5b98b-dcff-45b9-ac36-c3007953b774/alertmanager/0.log" Jul 22 01:18:27.038759 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:27.038718 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_80e5b98b-dcff-45b9-ac36-c3007953b774/config-reloader/0.log" Jul 22 01:18:27.065966 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:27.065931 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_80e5b98b-dcff-45b9-ac36-c3007953b774/kube-rbac-proxy-web/0.log" Jul 22 01:18:27.110705 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:27.110673 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_80e5b98b-dcff-45b9-ac36-c3007953b774/kube-rbac-proxy/0.log" Jul 22 01:18:27.144994 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:27.144963 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_80e5b98b-dcff-45b9-ac36-c3007953b774/kube-rbac-proxy-metric/0.log" Jul 22 01:18:27.170701 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:27.170670 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_80e5b98b-dcff-45b9-ac36-c3007953b774/prom-label-proxy/0.log" Jul 22 01:18:27.199949 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:27.199918 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_80e5b98b-dcff-45b9-ac36-c3007953b774/init-config-reloader/0.log" Jul 22 01:18:27.258204 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:27.258045 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-tbc4j_a7accd6c-8f60-4679-9ae1-331f59788c1b/cluster-monitoring-operator/0.log" Jul 22 01:18:27.519954 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:27.519923 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v7qfb_c0609636-2778-4018-aed2-68861437faa2/node-exporter/0.log" Jul 22 01:18:27.547641 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:27.547615 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v7qfb_c0609636-2778-4018-aed2-68861437faa2/kube-rbac-proxy/0.log" Jul 22 01:18:27.574860 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:27.574837 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v7qfb_c0609636-2778-4018-aed2-68861437faa2/init-textfile/0.log" Jul 22 01:18:27.695121 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:27.695090 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-f4pmz_f43e233f-b908-4f2c-aeaa-d9be509395b4/kube-rbac-proxy-main/0.log" Jul 22 01:18:27.719578 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:27.719545 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-f4pmz_f43e233f-b908-4f2c-aeaa-d9be509395b4/kube-rbac-proxy-self/0.log" Jul 22 01:18:27.741318 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:27.741278 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-f4pmz_f43e233f-b908-4f2c-aeaa-d9be509395b4/openshift-state-metrics/0.log" Jul 22 01:18:28.050592 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:28.050559 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79bddbf7-dgpjb_4ba8c1a9-42f9-4d53-8943-88f0b2a97e34/telemeter-client/0.log" Jul 22 01:18:28.074560 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:28.074535 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79bddbf7-dgpjb_4ba8c1a9-42f9-4d53-8943-88f0b2a97e34/reload/0.log" Jul 22 01:18:28.115202 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:28.115167 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79bddbf7-dgpjb_4ba8c1a9-42f9-4d53-8943-88f0b2a97e34/kube-rbac-proxy/0.log" Jul 22 01:18:29.358597 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.358562 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-zmtpr_e97ba97d-b165-4bd4-9a46-d8ac502f5d9d/networking-console-plugin/0.log" Jul 22 01:18:29.406173 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.406136 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc"] Jul 22 01:18:29.412007 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.411045 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.421528 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.421495 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc"] Jul 22 01:18:29.556665 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.556630 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjt8z\" (UniqueName: \"kubernetes.io/projected/4b983f05-398e-4d8d-aeb6-f361f4ce6513-kube-api-access-zjt8z\") pod \"perf-node-gather-daemonset-qh5hc\" (UID: \"4b983f05-398e-4d8d-aeb6-f361f4ce6513\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.556862 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.556672 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b983f05-398e-4d8d-aeb6-f361f4ce6513-podres\") pod \"perf-node-gather-daemonset-qh5hc\" (UID: \"4b983f05-398e-4d8d-aeb6-f361f4ce6513\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.556862 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.556767 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b983f05-398e-4d8d-aeb6-f361f4ce6513-sys\") pod \"perf-node-gather-daemonset-qh5hc\" (UID: \"4b983f05-398e-4d8d-aeb6-f361f4ce6513\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.556973 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.556876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b983f05-398e-4d8d-aeb6-f361f4ce6513-lib-modules\") pod \"perf-node-gather-daemonset-qh5hc\" (UID: \"4b983f05-398e-4d8d-aeb6-f361f4ce6513\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.557030 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.556990 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b983f05-398e-4d8d-aeb6-f361f4ce6513-proc\") pod \"perf-node-gather-daemonset-qh5hc\" (UID: \"4b983f05-398e-4d8d-aeb6-f361f4ce6513\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.657913 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.657821 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjt8z\" (UniqueName: \"kubernetes.io/projected/4b983f05-398e-4d8d-aeb6-f361f4ce6513-kube-api-access-zjt8z\") pod \"perf-node-gather-daemonset-qh5hc\" (UID: \"4b983f05-398e-4d8d-aeb6-f361f4ce6513\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.657913 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.657881 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b983f05-398e-4d8d-aeb6-f361f4ce6513-podres\") pod \"perf-node-gather-daemonset-qh5hc\" (UID: \"4b983f05-398e-4d8d-aeb6-f361f4ce6513\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.658142 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.657951 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b983f05-398e-4d8d-aeb6-f361f4ce6513-sys\") pod \"perf-node-gather-daemonset-qh5hc\" (UID: \"4b983f05-398e-4d8d-aeb6-f361f4ce6513\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.658142 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.657997 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b983f05-398e-4d8d-aeb6-f361f4ce6513-lib-modules\") pod \"perf-node-gather-daemonset-qh5hc\" (UID: \"4b983f05-398e-4d8d-aeb6-f361f4ce6513\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.658142 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.658079 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b983f05-398e-4d8d-aeb6-f361f4ce6513-podres\") pod \"perf-node-gather-daemonset-qh5hc\" (UID: \"4b983f05-398e-4d8d-aeb6-f361f4ce6513\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.658142 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.658089 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b983f05-398e-4d8d-aeb6-f361f4ce6513-proc\") pod \"perf-node-gather-daemonset-qh5hc\" (UID: \"4b983f05-398e-4d8d-aeb6-f361f4ce6513\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.658343 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.658144 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b983f05-398e-4d8d-aeb6-f361f4ce6513-proc\") pod \"perf-node-gather-daemonset-qh5hc\" (UID: \"4b983f05-398e-4d8d-aeb6-f361f4ce6513\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.658343 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.658241 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b983f05-398e-4d8d-aeb6-f361f4ce6513-sys\") pod \"perf-node-gather-daemonset-qh5hc\" (UID: \"4b983f05-398e-4d8d-aeb6-f361f4ce6513\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.658343 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.658271 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b983f05-398e-4d8d-aeb6-f361f4ce6513-lib-modules\") pod \"perf-node-gather-daemonset-qh5hc\" (UID: \"4b983f05-398e-4d8d-aeb6-f361f4ce6513\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.666972 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.666941 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjt8z\" (UniqueName: \"kubernetes.io/projected/4b983f05-398e-4d8d-aeb6-f361f4ce6513-kube-api-access-zjt8z\") pod \"perf-node-gather-daemonset-qh5hc\" (UID: \"4b983f05-398e-4d8d-aeb6-f361f4ce6513\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.725609 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.725576 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:29.944539 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:29.944511 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc"] Jul 22 01:18:29.945686 ip-10-0-132-178 kubenswrapper[2567]: W0722 01:18:29.945655 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4b983f05_398e_4d8d_aeb6_f361f4ce6513.slice/crio-f7c30220444a2bf6c5406c81cb09a696af6c4430d6b4dfce55a2603a95de9365 WatchSource:0}: Error finding container f7c30220444a2bf6c5406c81cb09a696af6c4430d6b4dfce55a2603a95de9365: Status 404 returned error can't find the container with id f7c30220444a2bf6c5406c81cb09a696af6c4430d6b4dfce55a2603a95de9365 Jul 22 01:18:30.047842 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:30.047813 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" event={"ID":"4b983f05-398e-4d8d-aeb6-f361f4ce6513","Type":"ContainerStarted","Data":"f7c30220444a2bf6c5406c81cb09a696af6c4430d6b4dfce55a2603a95de9365"} Jul 22 01:18:30.575842 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:30.575804 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-dqz2j_ae9fa646-a31d-473d-ab86-a537257cfe17/download-server/0.log" Jul 22 01:18:31.055793 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:31.055746 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" event={"ID":"4b983f05-398e-4d8d-aeb6-f361f4ce6513","Type":"ContainerStarted","Data":"8a4d5e59bd31b51a47cedf907823c8a6361365bdf102eb897a6a65ac5f9cf120"} Jul 22 01:18:31.055970 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:31.055827 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:31.077207 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:31.077147 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" podStartSLOduration=2.077127135 podStartE2EDuration="2.077127135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:18:31.076235057 +0000 UTC m=+3119.402307262" watchObservedRunningTime="2026-07-22 01:18:31.077127135 +0000 UTC m=+3119.403199338" Jul 22 01:18:31.181952 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:31.181920 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-qglr2_0db5ed04-ffe2-47d9-a85a-1934f3aeb751/volume-data-source-validator/0.log" Jul 22 01:18:32.332894 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:32.332857 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cdzlx_0f69f381-426d-422c-a9df-a007c2e4d0e6/dns/0.log" Jul 22 01:18:32.352988 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:32.352958 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cdzlx_0f69f381-426d-422c-a9df-a007c2e4d0e6/kube-rbac-proxy/0.log" Jul 22 01:18:32.402720 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:32.402683 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kxz6k_c28ec8cc-d99c-4ca5-9c43-4e0268ec46b5/dns-node-resolver/0.log" Jul 22 01:18:33.055896 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:33.055865 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lv725_d60d84ba-a0ae-4e46-9fdf-7830c7d5e798/node-ca/0.log" Jul 22 01:18:34.242171 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:34.242138 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6467ff7b5c-vxdwq_004b59a0-37e8-4897-9f04-58797d56b611/kube-auth-proxy/0.log" Jul 22 01:18:34.345868 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:34.345814 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xphnd_7fa3fb06-305b-4d41-91cc-ab7a360ef18f/payload-pre-processing/0.log" Jul 22 01:18:34.392204 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:34.392174 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-xx7xj_69b48eb0-702c-4da5-992d-07def03753ca/payload-processing/0.log" Jul 22 01:18:34.416482 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:34.416457 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7584584bbd-c77b7_09532874-5a40-44cd-b95b-1b3e9ad8ecfd/router/0.log" Jul 22 01:18:35.069252 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:35.069211 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5kzgh_2f80c51d-33fe-4bf1-8a2d-e6d0f2f0982a/serve-healthcheck-canary/0.log" Jul 22 01:18:35.783627 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:35.783596 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-sdhl6_eb290190-a2f9-4fd6-8267-b395887f767b/insights-operator/0.log" Jul 22 01:18:35.784977 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:35.784949 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-sdhl6_eb290190-a2f9-4fd6-8267-b395887f767b/insights-operator/1.log" Jul 22 01:18:35.886956 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:35.886925 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-96fng_a087a509-8906-4b67-a39d-bf24cd20c7da/kube-rbac-proxy/0.log" Jul 22 01:18:35.910384 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:35.910360 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-96fng_a087a509-8906-4b67-a39d-bf24cd20c7da/exporter/0.log" Jul 22 01:18:35.933441 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:35.933413 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-96fng_a087a509-8906-4b67-a39d-bf24cd20c7da/extractor/0.log" Jul 22 01:18:37.073173 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:37.073139 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-qh5hc" Jul 22 01:18:38.928031 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:38.927949 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb84b476f-vg4gh_c5e7758c-a583-43fc-b0d3-06855c41cbbd/manager/0.log" Jul 22 01:18:38.956625 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:38.956599 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb84b476f-vg4gh_c5e7758c-a583-43fc-b0d3-06855c41cbbd/copy-manifests/0.log" Jul 22 01:18:39.959719 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:39.959687 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6467ff7b5c-vxdwq_004b59a0-37e8-4897-9f04-58797d56b611/kube-auth-proxy/0.log" Jul 22 01:18:40.040153 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:40.040116 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xphnd_7fa3fb06-305b-4d41-91cc-ab7a360ef18f/payload-pre-processing/0.log" Jul 22 01:18:40.064338 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:40.064298 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-xx7xj_69b48eb0-702c-4da5-992d-07def03753ca/payload-processing/0.log" Jul 22 01:18:40.080595 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:40.080572 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7584584bbd-c77b7_09532874-5a40-44cd-b95b-1b3e9ad8ecfd/router/0.log" Jul 22 01:18:40.291323 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:40.291290 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7bd6f87dc6-pnvb8_1d17de76-1014-4177-b623-b04844ccb839/manager/0.log" Jul 22 01:18:41.456850 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:41.456816 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6467ff7b5c-vxdwq_004b59a0-37e8-4897-9f04-58797d56b611/kube-auth-proxy/0.log" Jul 22 01:18:41.539171 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:41.539138 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xphnd_7fa3fb06-305b-4d41-91cc-ab7a360ef18f/payload-pre-processing/0.log" Jul 22 01:18:41.569661 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:41.569575 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-xx7xj_69b48eb0-702c-4da5-992d-07def03753ca/payload-processing/0.log" Jul 22 01:18:41.586992 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:41.586960 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7584584bbd-c77b7_09532874-5a40-44cd-b95b-1b3e9ad8ecfd/router/0.log" Jul 22 01:18:45.816736 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:45.816696 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-kfnl7_f0b40554-c78f-463e-bce6-b4ac0b2b1c51/kube-storage-version-migrator-operator/1.log" Jul 22 01:18:45.818753 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:45.818723 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-kfnl7_f0b40554-c78f-463e-bce6-b4ac0b2b1c51/kube-storage-version-migrator-operator/0.log" Jul 22 01:18:46.762157 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:46.762130 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5pqtv_66d2c403-f38d-4c15-b761-8d50c9f3b142/kube-multus-additional-cni-plugins/0.log" Jul 22 01:18:46.782758 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:46.782740 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5pqtv_66d2c403-f38d-4c15-b761-8d50c9f3b142/egress-router-binary-copy/0.log" Jul 22 01:18:46.805291 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:46.805270 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5pqtv_66d2c403-f38d-4c15-b761-8d50c9f3b142/cni-plugins/0.log" Jul 22 01:18:46.826627 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:46.826607 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5pqtv_66d2c403-f38d-4c15-b761-8d50c9f3b142/bond-cni-plugin/0.log" Jul 22 01:18:46.849259 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:46.849187 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5pqtv_66d2c403-f38d-4c15-b761-8d50c9f3b142/routeoverride-cni/0.log" Jul 22 01:18:46.869888 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:46.869862 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5pqtv_66d2c403-f38d-4c15-b761-8d50c9f3b142/whereabouts-cni-bincopy/0.log" Jul 22 01:18:46.891621 ip-10-0-132-178 kubenswrapper[2567]: I0722 01:18:46.891603 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5pqtv_66d2c403-f38d-4c15-b761-8d50c9f3b142/whereabouts-cni/0.log"