Jul 09 18:51:27.069090 ip-10-0-133-185 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 09 18:51:27.069102 ip-10-0-133-185 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 09 18:51:27.069110 ip-10-0-133-185 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 09 18:51:27.069415 ip-10-0-133-185 systemd[1]: Failed to start Kubernetes Kubelet. Jul 09 18:51:37.175826 ip-10-0-133-185 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 09 18:51:37.175842 ip-10-0-133-185 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 354445d0daa34bca8a8fcbddf1d915f4 -- Jul 09 18:53:33.509520 ip-10-0-133-185 systemd[1]: Starting Kubernetes Kubelet... Jul 09 18:53:34.041786 ip-10-0-133-185 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 18:53:34.041786 ip-10-0-133-185 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 18:53:34.041786 ip-10-0-133-185 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 18:53:34.041786 ip-10-0-133-185 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 18:53:34.041786 ip-10-0-133-185 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 18:53:34.041786 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.898687 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 18:53:34.041786 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904517 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:53:34.041786 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904530 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:53:34.041786 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904536 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:53:34.041786 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904540 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:53:34.041786 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904543 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:53:34.041786 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904546 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:53:34.041786 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904549 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:53:34.041786 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904551 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:53:33.976995 ip-10-0-133-185 systemd[1]: Started Kubernetes Kubelet. Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904554 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904557 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904559 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904562 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904564 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904567 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904569 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904572 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904575 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904578 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904580 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904583 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904586 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904589 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904591 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904594 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904596 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904599 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904602 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:53:34.043544 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904605 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904607 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904611 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904613 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904616 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904619 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904622 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904624 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904627 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904629 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904632 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904634 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904637 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904639 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904642 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904644 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904647 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904649 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904652 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904654 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:53:34.044276 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904657 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904659 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904662 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904665 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904668 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904671 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904673 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904676 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904678 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904681 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904683 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904686 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904689 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904692 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904694 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904697 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904700 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904703 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904705 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904708 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:53:34.044856 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904710 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904713 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904716 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904718 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904721 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904724 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904726 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904729 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904731 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904737 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904741 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904744 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904747 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904755 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904758 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904761 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904763 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904767 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904771 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:53:34.045539 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904774 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904777 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904779 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904782 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904784 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904786 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904789 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904792 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904794 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904797 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904800 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904802 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.904805 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905577 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905584 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905586 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905589 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905591 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905594 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905597 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:53:34.271779 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905599 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905603 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905608 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905612 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905615 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905618 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905620 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905630 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905633 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905636 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905638 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905641 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905643 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905646 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905649 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905652 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905655 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905657 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905660 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905662 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:53:34.285421 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905665 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905669 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905672 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905675 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905678 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905681 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905684 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905686 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905690 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905692 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905695 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905697 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905700 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905702 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905705 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905707 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905710 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905713 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905716 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:53:34.286263 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905718 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905726 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905729 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905732 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905734 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905736 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905739 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905741 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905744 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905746 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905748 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905751 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905754 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905756 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905759 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905761 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905764 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905768 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905771 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905773 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:53:34.286945 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905776 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905778 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905781 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905783 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905786 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905788 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905791 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905793 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905796 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905798 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905801 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905803 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905806 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905808 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905817 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905821 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905826 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905829 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905831 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905834 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:53:34.287703 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905836 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905839 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905841 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905844 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905846 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905849 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905851 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905854 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905857 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905859 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905862 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905864 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.905867 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906339 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906345 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906348 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906352 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906355 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906357 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906360 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:53:34.288524 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906363 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906365 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906369 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906372 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906375 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906378 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906387 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906390 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906393 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906396 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906398 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906401 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906403 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906406 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906408 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906411 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906413 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906418 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906423 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906426 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:53:34.289541 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906429 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906431 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906434 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906436 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906439 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906441 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906444 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906446 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906449 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906451 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906459 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906462 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906465 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906467 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906470 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906474 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906479 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906482 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906484 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906489 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:53:34.290520 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906491 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906494 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906497 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906500 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906503 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906506 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906509 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906511 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906514 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906517 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906520 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906523 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906525 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906529 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906532 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906534 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906537 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906539 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906542 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906544 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:53:34.291121 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906546 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906549 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906551 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906554 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906556 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906559 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906561 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906563 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906568 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906570 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906573 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906575 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906578 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906581 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906583 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906586 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906588 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906591 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906593 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906595 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:53:34.291758 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906598 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906602 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906605 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.906607 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.907306 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.907313 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.907317 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.907320 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.907323 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.907326 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.907329 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.907331 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907396 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907403 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907410 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907414 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907418 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907422 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907425 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907430 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 18:53:34.292301 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907434 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907437 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907441 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907444 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907447 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907451 2569 flags.go:64] FLAG: --cgroup-root="" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907454 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907457 2569 flags.go:64] FLAG: --client-ca-file="" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907460 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907463 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907467 2569 flags.go:64] FLAG: --cluster-domain="" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907469 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907472 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907476 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907480 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907484 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907486 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907489 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907493 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907496 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907498 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907501 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907504 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907507 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 18:53:34.292870 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907511 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907514 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907517 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907520 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907523 2569 flags.go:64] FLAG: --enable-server="true" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907525 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907530 2569 flags.go:64] FLAG: --event-burst="100" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907533 2569 flags.go:64] FLAG: --event-qps="50" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907538 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907541 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907544 2569 flags.go:64] FLAG: --eviction-hard="" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907548 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907551 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907554 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907557 2569 flags.go:64] FLAG: --eviction-soft="" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907560 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907563 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907566 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907569 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907571 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907574 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907580 2569 flags.go:64] FLAG: --feature-gates="" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907584 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907587 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907590 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 18:53:34.293507 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907593 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907596 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907599 2569 flags.go:64] FLAG: --help="false" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907602 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-133-185.ec2.internal" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907606 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907608 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907611 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907615 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907618 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907621 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907624 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907627 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907629 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907632 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907635 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907639 2569 flags.go:64] FLAG: --kube-reserved="" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907642 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907645 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907648 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907651 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907654 2569 flags.go:64] FLAG: --lock-file="" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907656 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907660 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907663 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 18:53:34.294238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907668 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907671 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907674 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907676 2569 flags.go:64] FLAG: --logging-format="text" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907680 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907684 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907686 2569 flags.go:64] FLAG: --manifest-url="" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907689 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907693 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907696 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907700 2569 flags.go:64] FLAG: --max-pods="110" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907703 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907706 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907709 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907712 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907715 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907718 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907721 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907727 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907730 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907733 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907736 2569 flags.go:64] FLAG: --pod-cidr="" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907739 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 09 18:53:34.294872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907745 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907749 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907752 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907755 2569 flags.go:64] FLAG: --port="10250" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907758 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907760 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0db8a16addf15429f" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907764 2569 flags.go:64] FLAG: --qos-reserved="" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907767 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907770 2569 flags.go:64] FLAG: --register-node="true" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907773 2569 flags.go:64] FLAG: --register-with-taints="" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907776 2569 flags.go:64] FLAG: --registry-burst="10" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907780 2569 flags.go:64] FLAG: --registry-qps="5" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907782 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907785 2569 flags.go:64] FLAG: --reserved-memory="" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907790 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907793 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907796 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907799 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907802 2569 flags.go:64] FLAG: --runonce="false" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907805 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907808 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907811 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907814 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907817 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907820 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907823 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 18:53:34.295576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907826 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907829 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907832 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907834 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907837 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907840 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907845 2569 flags.go:64] FLAG: --system-cgroups="" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907847 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907852 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907855 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907857 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907862 2569 flags.go:64] FLAG: --tls-min-version="" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907864 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907867 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907870 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907873 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907876 2569 flags.go:64] FLAG: --v="2" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907880 2569 flags.go:64] FLAG: --version="false" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907884 2569 flags.go:64] FLAG: --vmodule="" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907887 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.907891 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908028 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908032 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908035 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:53:34.296584 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908038 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908040 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908043 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908046 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908048 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908051 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908054 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908056 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908059 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908062 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908064 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908067 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908069 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908072 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908077 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908080 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908083 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908085 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908088 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:53:34.297266 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908090 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908093 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908095 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908098 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908101 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908107 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908110 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908112 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908115 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908119 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908122 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908124 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908127 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908129 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908132 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908134 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908137 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908139 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908142 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908145 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:53:36.085096 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908147 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908150 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908152 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908155 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908158 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908160 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908164 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908169 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908172 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908175 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908178 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908181 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908185 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908188 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908191 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908193 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908196 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908199 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908202 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:53:36.086024 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908205 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908208 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908211 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908214 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908217 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908239 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908242 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908244 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908247 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908250 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908253 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908256 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908258 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908261 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908264 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908266 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908269 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908271 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908274 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908277 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:53:36.086622 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908281 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908284 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908286 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908289 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908291 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908294 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908296 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908299 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908301 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908304 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908307 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908309 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908312 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908315 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908319 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908322 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908324 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:53:36.087243 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.908327 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:53:36.087714 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.909244 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 18:53:36.087714 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.918478 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 09 18:53:36.087714 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.918496 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 18:53:36.087714 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.918511 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 18:53:36.087714 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.918517 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 18:53:36.087714 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918582 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:53:36.087714 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918587 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:53:36.087714 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918592 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:53:36.087714 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918596 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:53:36.087714 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918599 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:53:36.087714 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918604 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:53:36.087714 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918607 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:53:36.087714 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918610 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:53:36.087714 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918614 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:53:36.087714 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918617 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918620 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918623 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918629 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918633 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918640 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918644 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918649 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918652 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918655 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918659 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918662 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918665 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918668 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918673 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918676 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918678 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918683 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918686 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918689 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:53:36.088181 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918691 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918696 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918699 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918703 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918706 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918708 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918711 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918714 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918717 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918719 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918722 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918726 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918728 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918731 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918733 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918738 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918740 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918745 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918750 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:53:36.088707 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918753 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918756 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918760 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918763 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918768 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918773 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918777 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918783 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918786 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918789 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918794 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918797 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918800 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918802 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918807 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918810 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918816 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918820 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918829 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:53:36.089284 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918832 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918835 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918839 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918842 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918844 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918849 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918852 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918855 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918857 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918861 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918864 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918867 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918869 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918875 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918877 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918880 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918885 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918888 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918891 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918895 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:53:36.089799 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918899 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:53:36.090371 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918904 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:53:36.090371 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918906 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:53:36.090371 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918909 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:53:36.090371 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918911 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:53:36.090371 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918914 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:53:36.090371 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918916 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:53:36.090371 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918919 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:53:36.090371 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918921 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:53:36.090371 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918925 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:53:36.090371 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918928 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:53:36.090371 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.918932 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:53:36.090371 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.918938 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 18:53:36.090371 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919079 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:53:36.090371 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919086 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:53:36.090371 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919089 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919094 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919098 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919101 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919105 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919108 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919110 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919113 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919116 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919118 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919121 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919125 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919129 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919132 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919137 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919142 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919147 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919150 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919162 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:53:36.090784 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919165 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919167 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919172 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919175 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919177 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919181 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919184 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919187 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919189 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919195 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919197 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919200 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919204 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919207 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919210 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919216 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919232 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919242 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919245 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919248 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:53:36.091363 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919252 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919255 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919258 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919262 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919265 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919267 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919270 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919272 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919277 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919280 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919282 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919285 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919291 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919294 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919297 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919300 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919302 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919304 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919307 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919310 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:53:36.091881 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919314 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919318 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919320 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919323 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919325 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919328 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919330 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919333 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919335 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919338 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919340 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919343 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919345 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919348 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919350 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919354 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919357 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919360 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919363 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919365 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:53:36.092463 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919369 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919372 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919374 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919377 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919379 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919382 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919385 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919387 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919390 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919392 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919395 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919397 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919400 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919404 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919406 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919409 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919411 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:53:36.093054 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:33.919414 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:53:36.093540 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.919418 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 18:53:36.093540 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.919522 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 18:53:36.093540 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.921508 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 18:53:36.093540 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.922781 2569 server.go:1019] "Starting client certificate rotation" Jul 09 18:53:36.093540 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.922885 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 18:53:36.093540 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.922918 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 18:53:36.093540 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.925098 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 18:53:36.093540 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.931981 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 18:53:36.093540 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.940462 2569 log.go:25] "Validated CRI v1 runtime API" Jul 09 18:53:36.093540 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.942428 2569 log.go:25] "Validated CRI v1 image API" Jul 09 18:53:36.093540 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.947033 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 18:53:36.093540 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.949300 2569 fs.go:135] Filesystem UUIDs: map[76cd8526-b0a5-4a7a-9e44-c3326bb1cf07:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d115a0e7-935f-4b7d-92c9-4a0a70e1084d:/dev/nvme0n1p3] Jul 09 18:53:36.093882 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.949314 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 18:53:36.094002 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.953444 2569 manager.go:216] Machine: {Timestamp:2026-07-09 18:53:33.952940375 +0000 UTC m=+0.331827920 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101151 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec255db437b1f70e6f4399a0deedd723 SystemUUID:ec255db4-37b1-f70e-6f43-99a0deedd723 BootID:354445d0-daa3-4bca-8a8f-cbddf1d915f4 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5d:c1:f9:7c:a7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5d:c1:f9:7c:a7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:8f:4b:b2:24:5c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 18:53:36.094002 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.953548 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 18:53:36.094002 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.953623 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 18:53:36.094002 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.956332 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 18:53:36.094002 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.956359 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-185.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 18:53:36.094002 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.957849 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 18:53:36.094002 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.957858 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 18:53:36.094002 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.957866 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.958754 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.959980 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.960304 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.963626 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.963740 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.963767 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.963777 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.963787 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.963796 2569 kubelet.go:398] "Adding apiserver pod source" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.963803 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.964102 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.964115 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.966241 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.967959 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.967987 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.968124 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.968135 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.968141 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.968147 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.968154 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.968159 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.969139 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.969145 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.969151 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.969157 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.969170 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.969178 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.970025 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.970035 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.973445 2569 server.go:1268] "Started kubelet" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.973480 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.973564 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.973627 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.977974 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.978490 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:33.978756 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.980062 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-185.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:33.980555 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 18:53:36.094276 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:33.980677 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-185.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:33.980195 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-185.ec2.internal.18c0b3dcf2031bd7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-185.ec2.internal,UID:ip-10-0-133-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-185.ec2.internal,},FirstTimestamp:2026-07-09 18:53:33.973183447 +0000 UTC m=+0.352070981,LastTimestamp:2026-07-09 18:53:33.973183447 +0000 UTC m=+0.352070981,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-185.ec2.internal,}" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.985686 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.985698 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:33.985809 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.986338 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.986352 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.986338 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.986439 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.986445 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.986843 2569 factory.go:55] Registering systemd factory Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.986866 2569 factory.go:223] Registration of the systemd container factory successfully Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.987043 2569 factory.go:153] Registering CRI-O factory Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.987053 2569 factory.go:223] Registration of the crio container factory successfully Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.987092 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.987114 2569 factory.go:103] Registering Raw factory Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.987130 2569 manager.go:1196] Started watching for new ooms in manager Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.987553 2569 manager.go:349] Starting recovery of all containers Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:33.987752 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-185.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.996305 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jt9cg" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:33.997160 2569 manager.go:354] Recovery completed Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:33.998135 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.001741 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.003371 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.003399 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.003413 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.003865 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.003874 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.003887 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.003918 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.004481 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.004498 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.004507 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:53:36.095359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.005236 2569 policy_none.go:49] "None policy: Start" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.005250 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.005260 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.006302 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jt9cg" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.006319 2569 policy_none.go:47] "Start" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.037770 2569 manager.go:337] "Starting Device Plugin manager" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:34.037804 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.037817 2569 server.go:87] "Starting device plugin registration server" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.038044 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.038056 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.038129 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.038188 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.038194 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:34.038635 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:34.038681 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.133111 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.134275 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.134308 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.134329 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:34.134361 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.137348 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.138350 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.139098 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.139132 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.139145 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.139171 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.147592 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:34.147612 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-185.ec2.internal\": node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:34.173639 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.234675 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-185.ec2.internal"] Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.234731 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.235563 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.235593 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.235604 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.236778 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.236903 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.236933 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.237470 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:53:36.096481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.237491 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.237505 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.237532 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.237552 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.237567 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.238923 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.238944 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.241628 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.241643 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.241652 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:34.264911 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-185.ec2.internal\" not found" node="ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:34.269301 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-185.ec2.internal\" not found" node="ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:34.274314 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.287700 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dcabe690d34269c2f19f5d969bbfccfb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal\" (UID: \"dcabe690d34269c2f19f5d969bbfccfb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.287722 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcabe690d34269c2f19f5d969bbfccfb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal\" (UID: \"dcabe690d34269c2f19f5d969bbfccfb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:34.287748 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58be071d26cafa55b0bf4c584a43070e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-185.ec2.internal\" (UID: \"58be071d26cafa55b0bf4c584a43070e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:35.928649 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:35.942297 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dcabe690d34269c2f19f5d969bbfccfb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal\" (UID: \"dcabe690d34269c2f19f5d969bbfccfb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:35.942321 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcabe690d34269c2f19f5d969bbfccfb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal\" (UID: \"dcabe690d34269c2f19f5d969bbfccfb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:35.942338 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58be071d26cafa55b0bf4c584a43070e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-185.ec2.internal\" (UID: \"58be071d26cafa55b0bf4c584a43070e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.097640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:35.942381 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58be071d26cafa55b0bf4c584a43070e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-185.ec2.internal\" (UID: \"58be071d26cafa55b0bf4c584a43070e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.242126 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:35.942403 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dcabe690d34269c2f19f5d969bbfccfb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal\" (UID: \"dcabe690d34269c2f19f5d969bbfccfb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.242126 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:35.942409 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcabe690d34269c2f19f5d969bbfccfb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal\" (UID: \"dcabe690d34269c2f19f5d969bbfccfb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.242126 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:36.029360 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:36.242126 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:36.120832 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.242126 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:36.125705 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-185.ec2.internal" Jul 09 18:53:36.242126 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:36.129819 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:36.242126 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:36.230360 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:36.435561 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:36.330842 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:36.435561 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:36.431504 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:36.601291 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:36.476241 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 18:53:36.601291 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:36.476364 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 18:53:36.601291 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:36.531607 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:36.601291 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:36.563328 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 18:48:35 +0000 UTC" deadline="2028-02-07 08:11:33.041265884 +0000 UTC" Jul 09 18:53:36.601291 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:36.563352 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13861h17m56.477916599s" Jul 09 18:53:36.752193 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:36.632724 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:36.752193 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:36.733155 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:36.871133 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:36.833585 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:37.027580 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:36.878378 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58be071d26cafa55b0bf4c584a43070e.slice/crio-b07e5fb659e23458b1633630d0682d4e9923d6ea5170ee54aedbb93cf03098a5 WatchSource:0}: Error finding container b07e5fb659e23458b1633630d0682d4e9923d6ea5170ee54aedbb93cf03098a5: Status 404 returned error can't find the container with id b07e5fb659e23458b1633630d0682d4e9923d6ea5170ee54aedbb93cf03098a5 Jul 09 18:53:37.027580 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:36.879686 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcabe690d34269c2f19f5d969bbfccfb.slice/crio-07e83c147c57f89856b1ece5c08d19bf04180758f1787187a676542155dfad6a WatchSource:0}: Error finding container 07e83c147c57f89856b1ece5c08d19bf04180758f1787187a676542155dfad6a: Status 404 returned error can't find the container with id 07e83c147c57f89856b1ece5c08d19bf04180758f1787187a676542155dfad6a Jul 09 18:53:37.027580 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:36.884816 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 18:53:37.027580 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:36.933771 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:37.027580 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.007197 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:53:37.187100 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:37.034376 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-185.ec2.internal\" not found" Jul 09 18:53:37.187100 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.125201 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:53:37.187100 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.133561 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:53:37.187100 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.140545 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal" Jul 09 18:53:37.187100 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.155076 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 18:53:37.187100 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.156228 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-185.ec2.internal" Jul 09 18:53:37.187100 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.166371 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.518594 2569 apiserver.go:52] "Watching apiserver" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.527077 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.528391 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-9q928","openshift-multus/multus-xxb68","openshift-network-diagnostics/network-check-target-4hsrp","kube-system/konnectivity-agent-lmbqh","kube-system/kube-apiserver-proxy-ip-10-0-133-185.ec2.internal","openshift-dns/node-resolver-v6rsk","openshift-multus/network-metrics-daemon-kfvdz","openshift-network-operator/iptables-alerter-gmh5g","openshift-ovn-kubernetes/ovnkube-node-gsr6w","kube-system/global-pull-secret-syncer-kss86","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt","openshift-cluster-node-tuning-operator/tuned-s4jwv","openshift-image-registry/node-ca-bnbtq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal"] Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.530407 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:37.530481 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.531640 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.533189 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:37.533254 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.533941 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.534078 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.534304 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lmbqh" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.534453 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.534690 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jt29m\"" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.535498 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gmh5g" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.536681 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.536871 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v6rsk" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.537295 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zwhcx\"" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.537352 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.538116 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.538375 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.538700 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rwldj\"" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.538979 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.539160 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.539177 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.539624 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bnbtq" Jul 09 18:53:37.576492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.540425 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.540659 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.540670 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lmw5k\"" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.540882 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.541092 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.541311 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wdmng\"" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.541676 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.541694 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.541921 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.543149 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.543514 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.543963 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.544196 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qmfjd\"" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.545889 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.546903 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:37.546984 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.547655 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.548067 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.548115 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551157 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs\") pod \"network-metrics-daemon-kfvdz\" (UID: \"c46a7770-1771-4f8c-b35b-33ed971625c4\") " pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551191 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-kubelet\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551215 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-cni-netd\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551237 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qljjw\" (UniqueName: \"kubernetes.io/projected/348e3292-41e0-44d6-b59b-81372e7d0b81-kube-api-access-qljjw\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.647976 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551261 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-cnibin\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.649572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551286 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-multus-socket-dir-parent\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.649572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551308 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-var-lib-openvswitch\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.649572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551331 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-registration-dir\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.649572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551354 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-cni-bin\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.649572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551383 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab56b93f-4a0a-4f46-9910-94df2f70621f-host\") pod \"node-ca-bnbtq\" (UID: \"ab56b93f-4a0a-4f46-9910-94df2f70621f\") " pod="openshift-image-registry/node-ca-bnbtq" Jul 09 18:53:37.649572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551405 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-cni-binary-copy\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.649572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551426 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-run-multus-certs\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.649572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551462 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dfcb4a64-02f6-451c-97c0-451804e2bd36-hosts-file\") pod \"node-resolver-v6rsk\" (UID: \"dfcb4a64-02f6-451c-97c0-451804e2bd36\") " pod="openshift-dns/node-resolver-v6rsk" Jul 09 18:53:37.649572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551482 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rpk6\" (UniqueName: \"kubernetes.io/projected/89a666fa-9965-49a8-930b-7fcebff2033b-kube-api-access-2rpk6\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.649572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551502 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-os-release\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.649572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551524 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1e300ac-d943-40a5-847c-f33b16c01ab8-iptables-alerter-script\") pod \"iptables-alerter-gmh5g\" (UID: \"a1e300ac-d943-40a5-847c-f33b16c01ab8\") " pod="openshift-network-operator/iptables-alerter-gmh5g" Jul 09 18:53:37.649572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551578 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz65d\" (UniqueName: \"kubernetes.io/projected/dfcb4a64-02f6-451c-97c0-451804e2bd36-kube-api-access-xz65d\") pod \"node-resolver-v6rsk\" (UID: \"dfcb4a64-02f6-451c-97c0-451804e2bd36\") " pod="openshift-dns/node-resolver-v6rsk" Jul 09 18:53:37.649572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551602 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-run-openvswitch\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.649572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551627 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-node-log\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.650213 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551649 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-var-lib-cni-bin\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.650213 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551671 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfcb4a64-02f6-451c-97c0-451804e2bd36-tmp-dir\") pod \"node-resolver-v6rsk\" (UID: \"dfcb4a64-02f6-451c-97c0-451804e2bd36\") " pod="openshift-dns/node-resolver-v6rsk" Jul 09 18:53:37.650213 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551694 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-log-socket\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.650213 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551717 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-var-lib-kubelet\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.650213 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551754 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w6fj\" (UniqueName: \"kubernetes.io/projected/a1e300ac-d943-40a5-847c-f33b16c01ab8-kube-api-access-6w6fj\") pod \"iptables-alerter-gmh5g\" (UID: \"a1e300ac-d943-40a5-847c-f33b16c01ab8\") " pod="openshift-network-operator/iptables-alerter-gmh5g" Jul 09 18:53:37.650213 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551781 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vsgt\" (UniqueName: \"kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt\") pod \"network-check-target-4hsrp\" (UID: \"3f612d36-6ce3-44a2-9a90-61dd316a6703\") " pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:37.650213 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551805 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n52n\" (UniqueName: \"kubernetes.io/projected/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-kube-api-access-9n52n\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.650213 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551846 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-run-systemd\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.650213 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551896 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-device-dir\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.650213 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.551990 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/348e3292-41e0-44d6-b59b-81372e7d0b81-ovn-node-metrics-cert\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.650213 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552017 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-multus-cni-dir\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.650213 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552053 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-hostroot\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.650213 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552088 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-multus-daemon-config\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.650213 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552114 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-sys-fs\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.651510 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552136 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.651510 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552166 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-etc-kubernetes\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.651510 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552187 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-slash\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.651510 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552208 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.651510 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552230 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-multus-conf-dir\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.651510 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552271 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd7d71e8-6272-469f-8569-7d97ac86c975-konnectivity-ca\") pod \"konnectivity-agent-lmbqh\" (UID: \"dd7d71e8-6272-469f-8569-7d97ac86c975\") " pod="kube-system/konnectivity-agent-lmbqh" Jul 09 18:53:37.651510 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-etc-selinux\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.651510 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552314 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab56b93f-4a0a-4f46-9910-94df2f70621f-serviceca\") pod \"node-ca-bnbtq\" (UID: \"ab56b93f-4a0a-4f46-9910-94df2f70621f\") " pod="openshift-image-registry/node-ca-bnbtq" Jul 09 18:53:37.651510 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552335 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-var-lib-cni-multus\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.651510 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552357 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-run-ovn\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.651510 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552379 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2znm\" (UniqueName: \"kubernetes.io/projected/ab56b93f-4a0a-4f46-9910-94df2f70621f-kube-api-access-b2znm\") pod \"node-ca-bnbtq\" (UID: \"ab56b93f-4a0a-4f46-9910-94df2f70621f\") " pod="openshift-image-registry/node-ca-bnbtq" Jul 09 18:53:37.651510 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552400 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-system-cni-dir\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.651510 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552422 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1e300ac-d943-40a5-847c-f33b16c01ab8-host-slash\") pod \"iptables-alerter-gmh5g\" (UID: \"a1e300ac-d943-40a5-847c-f33b16c01ab8\") " pod="openshift-network-operator/iptables-alerter-gmh5g" Jul 09 18:53:37.651510 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552441 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kcvxf\"" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552475 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-systemd-units\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552498 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-run-netns\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552524 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-run-netns\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552535 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552547 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l8br\" (UniqueName: \"kubernetes.io/projected/c46a7770-1771-4f8c-b35b-33ed971625c4-kube-api-access-4l8br\") pod \"network-metrics-daemon-kfvdz\" (UID: \"c46a7770-1771-4f8c-b35b-33ed971625c4\") " pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552570 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/348e3292-41e0-44d6-b59b-81372e7d0b81-ovnkube-config\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552591 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/348e3292-41e0-44d6-b59b-81372e7d0b81-env-overrides\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552605 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552621 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-run-k8s-cni-cncf-io\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552650 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd7d71e8-6272-469f-8569-7d97ac86c975-agent-certs\") pod \"konnectivity-agent-lmbqh\" (UID: \"dd7d71e8-6272-469f-8569-7d97ac86c975\") " pod="kube-system/konnectivity-agent-lmbqh" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552680 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552709 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-socket-dir\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552741 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-etc-openvswitch\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552765 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/348e3292-41e0-44d6-b59b-81372e7d0b81-ovnkube-script-lib\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552836 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 18:53:37.652167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.552911 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t6q6f\"" Jul 09 18:53:37.653322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.553032 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 18:53:37.653322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.553229 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 18:53:37.653322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.553856 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 18:53:37.653322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.554464 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fctnq\"" Jul 09 18:53:37.653322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.554642 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 18:53:37.653322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.556425 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 18:53:37.653322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.560827 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 18:53:37.653322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.588996 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-knbqz" Jul 09 18:53:37.653322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.600740 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-knbqz" Jul 09 18:53:37.653322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.642487 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 18:53:37.654547 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.653937 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.654547 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.653992 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-multus-conf-dir\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.654547 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654018 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd7d71e8-6272-469f-8569-7d97ac86c975-konnectivity-ca\") pod \"konnectivity-agent-lmbqh\" (UID: \"dd7d71e8-6272-469f-8569-7d97ac86c975\") " pod="kube-system/konnectivity-agent-lmbqh" Jul 09 18:53:37.654547 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654041 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-etc-selinux\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.654547 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654060 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab56b93f-4a0a-4f46-9910-94df2f70621f-serviceca\") pod \"node-ca-bnbtq\" (UID: \"ab56b93f-4a0a-4f46-9910-94df2f70621f\") " pod="openshift-image-registry/node-ca-bnbtq" Jul 09 18:53:37.654547 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654081 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-var-lib-cni-multus\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.654547 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654105 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-run-ovn\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.654547 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654123 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2znm\" (UniqueName: \"kubernetes.io/projected/ab56b93f-4a0a-4f46-9910-94df2f70621f-kube-api-access-b2znm\") pod \"node-ca-bnbtq\" (UID: \"ab56b93f-4a0a-4f46-9910-94df2f70621f\") " pod="openshift-image-registry/node-ca-bnbtq" Jul 09 18:53:37.654547 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654141 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-system-cni-dir\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.654547 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654158 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1e300ac-d943-40a5-847c-f33b16c01ab8-host-slash\") pod \"iptables-alerter-gmh5g\" (UID: \"a1e300ac-d943-40a5-847c-f33b16c01ab8\") " pod="openshift-network-operator/iptables-alerter-gmh5g" Jul 09 18:53:37.654547 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:37.654547 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654200 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-systemd\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.654547 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654217 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-tmp\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.654547 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654235 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-systemd-units\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.656743 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654253 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-run-netns\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.656743 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654271 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-run-netns\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.656743 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654293 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddf405a2-ba6c-4760-85e1-de6d4b699226-cnibin\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.656743 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654316 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l8br\" (UniqueName: \"kubernetes.io/projected/c46a7770-1771-4f8c-b35b-33ed971625c4-kube-api-access-4l8br\") pod \"network-metrics-daemon-kfvdz\" (UID: \"c46a7770-1771-4f8c-b35b-33ed971625c4\") " pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:37.656743 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654334 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/348e3292-41e0-44d6-b59b-81372e7d0b81-ovnkube-config\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.656743 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654369 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/348e3292-41e0-44d6-b59b-81372e7d0b81-env-overrides\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.656743 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654387 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-run-k8s-cni-cncf-io\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.656743 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654409 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd7d71e8-6272-469f-8569-7d97ac86c975-agent-certs\") pod \"konnectivity-agent-lmbqh\" (UID: \"dd7d71e8-6272-469f-8569-7d97ac86c975\") " pod="kube-system/konnectivity-agent-lmbqh" Jul 09 18:53:37.656743 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654428 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddf405a2-ba6c-4760-85e1-de6d4b699226-system-cni-dir\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.656743 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654446 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.656743 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654465 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-socket-dir\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.656743 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654484 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-etc-openvswitch\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.656743 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654502 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/348e3292-41e0-44d6-b59b-81372e7d0b81-ovnkube-script-lib\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.656743 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654524 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs\") pod \"network-metrics-daemon-kfvdz\" (UID: \"c46a7770-1771-4f8c-b35b-33ed971625c4\") " pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:37.657492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654548 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-kubelet\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.657492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654579 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-cni-netd\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.657492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654610 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-multus-conf-dir\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.657492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qljjw\" (UniqueName: \"kubernetes.io/projected/348e3292-41e0-44d6-b59b-81372e7d0b81-kube-api-access-qljjw\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.657492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654642 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-cnibin\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.657492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654694 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-multus-socket-dir-parent\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.657492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654725 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-kubernetes\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.657492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654751 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-var-lib-openvswitch\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.657492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654779 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-948cm\" (UniqueName: \"kubernetes.io/projected/ddf405a2-ba6c-4760-85e1-de6d4b699226-kube-api-access-948cm\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.657492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654803 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-sysconfig\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.657492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654828 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-registration-dir\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.657492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654852 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-cni-bin\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.657492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654876 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab56b93f-4a0a-4f46-9910-94df2f70621f-host\") pod \"node-ca-bnbtq\" (UID: \"ab56b93f-4a0a-4f46-9910-94df2f70621f\") " pod="openshift-image-registry/node-ca-bnbtq" Jul 09 18:53:37.657492 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654898 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-cni-binary-copy\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.658118 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654925 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-run-multus-certs\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.658118 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654966 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-modprobe-d\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.658118 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.654991 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-lib-modules\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.658118 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655014 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dfcb4a64-02f6-451c-97c0-451804e2bd36-hosts-file\") pod \"node-resolver-v6rsk\" (UID: \"dfcb4a64-02f6-451c-97c0-451804e2bd36\") " pod="openshift-dns/node-resolver-v6rsk" Jul 09 18:53:37.658118 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655040 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rpk6\" (UniqueName: \"kubernetes.io/projected/89a666fa-9965-49a8-930b-7fcebff2033b-kube-api-access-2rpk6\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.658118 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655067 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-os-release\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.658118 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655095 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1e300ac-d943-40a5-847c-f33b16c01ab8-iptables-alerter-script\") pod \"iptables-alerter-gmh5g\" (UID: \"a1e300ac-d943-40a5-847c-f33b16c01ab8\") " pod="openshift-network-operator/iptables-alerter-gmh5g" Jul 09 18:53:37.658118 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655121 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ddf405a2-ba6c-4760-85e1-de6d4b699226-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.658118 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655146 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-dbus\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:37.658118 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655163 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd7d71e8-6272-469f-8569-7d97ac86c975-konnectivity-ca\") pod \"konnectivity-agent-lmbqh\" (UID: \"dd7d71e8-6272-469f-8569-7d97ac86c975\") " pod="kube-system/konnectivity-agent-lmbqh" Jul 09 18:53:37.658118 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-sysctl-d\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.658118 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655193 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-tuned\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.658118 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz65d\" (UniqueName: \"kubernetes.io/projected/dfcb4a64-02f6-451c-97c0-451804e2bd36-kube-api-access-xz65d\") pod \"node-resolver-v6rsk\" (UID: \"dfcb4a64-02f6-451c-97c0-451804e2bd36\") " pod="openshift-dns/node-resolver-v6rsk" Jul 09 18:53:37.658118 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655244 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-run-openvswitch\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.658631 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655271 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-node-log\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.658631 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655295 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-var-lib-cni-bin\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.658631 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655317 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-run\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.658631 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655340 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-host\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.658631 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655383 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfcb4a64-02f6-451c-97c0-451804e2bd36-tmp-dir\") pod \"node-resolver-v6rsk\" (UID: \"dfcb4a64-02f6-451c-97c0-451804e2bd36\") " pod="openshift-dns/node-resolver-v6rsk" Jul 09 18:53:37.658631 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655408 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-log-socket\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.658631 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655427 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-var-lib-kubelet\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.658631 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655448 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w6fj\" (UniqueName: \"kubernetes.io/projected/a1e300ac-d943-40a5-847c-f33b16c01ab8-kube-api-access-6w6fj\") pod \"iptables-alerter-gmh5g\" (UID: \"a1e300ac-d943-40a5-847c-f33b16c01ab8\") " pod="openshift-network-operator/iptables-alerter-gmh5g" Jul 09 18:53:37.658631 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655470 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-var-lib-kubelet\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.658631 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655490 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zdfv\" (UniqueName: \"kubernetes.io/projected/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-kube-api-access-2zdfv\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.658631 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655515 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsgt\" (UniqueName: \"kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt\") pod \"network-check-target-4hsrp\" (UID: \"3f612d36-6ce3-44a2-9a90-61dd316a6703\") " pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:37.658631 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655534 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n52n\" (UniqueName: \"kubernetes.io/projected/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-kube-api-access-9n52n\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.658631 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655557 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-kubelet-config\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:37.658631 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655577 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ab56b93f-4a0a-4f46-9910-94df2f70621f-serviceca\") pod \"node-ca-bnbtq\" (UID: \"ab56b93f-4a0a-4f46-9910-94df2f70621f\") " pod="openshift-image-registry/node-ca-bnbtq" Jul 09 18:53:37.659143 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655582 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-run-systemd\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.659143 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddf405a2-ba6c-4760-85e1-de6d4b699226-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.659143 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655622 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-var-lib-cni-multus\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.659143 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655631 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-sysctl-conf\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.659143 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655657 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-device-dir\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.659143 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655664 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-run-ovn\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.659143 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655679 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/348e3292-41e0-44d6-b59b-81372e7d0b81-ovn-node-metrics-cert\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.659143 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655702 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-multus-cni-dir\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.659143 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655706 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab56b93f-4a0a-4f46-9910-94df2f70621f-host\") pod \"node-ca-bnbtq\" (UID: \"ab56b93f-4a0a-4f46-9910-94df2f70621f\") " pod="openshift-image-registry/node-ca-bnbtq" Jul 09 18:53:37.659143 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655737 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-hostroot\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.659143 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655753 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-system-cni-dir\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.659143 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655759 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-multus-daemon-config\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.659143 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655781 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ddf405a2-ba6c-4760-85e1-de6d4b699226-os-release\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.659143 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655788 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1e300ac-d943-40a5-847c-f33b16c01ab8-host-slash\") pod \"iptables-alerter-gmh5g\" (UID: \"a1e300ac-d943-40a5-847c-f33b16c01ab8\") " pod="openshift-network-operator/iptables-alerter-gmh5g" Jul 09 18:53:37.659653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655802 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddf405a2-ba6c-4760-85e1-de6d4b699226-cni-binary-copy\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.659653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-sys\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.659653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655843 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-systemd-units\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.659653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655854 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-sys-fs\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.659653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655879 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.659653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655881 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-run-netns\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.659653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655903 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-etc-kubernetes\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.659653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655919 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-run-netns\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.659653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655923 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ddf405a2-ba6c-4760-85e1-de6d4b699226-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.659653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-slash\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.659653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.656040 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-slash\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.659653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.656391 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-cni-binary-copy\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.659653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.656443 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-run-multus-certs\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.659653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.656506 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dfcb4a64-02f6-451c-97c0-451804e2bd36-hosts-file\") pod \"node-resolver-v6rsk\" (UID: \"dfcb4a64-02f6-451c-97c0-451804e2bd36\") " pod="openshift-dns/node-resolver-v6rsk" Jul 09 18:53:37.659653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.656717 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-os-release\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.660252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.656719 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/348e3292-41e0-44d6-b59b-81372e7d0b81-ovnkube-config\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.660252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.657093 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/348e3292-41e0-44d6-b59b-81372e7d0b81-env-overrides\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.660252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.657151 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-run-k8s-cni-cncf-io\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.660252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.657242 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1e300ac-d943-40a5-847c-f33b16c01ab8-iptables-alerter-script\") pod \"iptables-alerter-gmh5g\" (UID: \"a1e300ac-d943-40a5-847c-f33b16c01ab8\") " pod="openshift-network-operator/iptables-alerter-gmh5g" Jul 09 18:53:37.660252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.657437 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-run-systemd\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.660252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.657485 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-run-openvswitch\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.660252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.657533 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-node-log\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.660252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.657574 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-device-dir\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.660252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.655274 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-etc-selinux\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.660252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.657622 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-var-lib-cni-bin\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.660252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.657849 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfcb4a64-02f6-451c-97c0-451804e2bd36-tmp-dir\") pod \"node-resolver-v6rsk\" (UID: \"dfcb4a64-02f6-451c-97c0-451804e2bd36\") " pod="openshift-dns/node-resolver-v6rsk" Jul 09 18:53:37.660252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.657882 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-multus-cni-dir\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.660252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.657924 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-host-var-lib-kubelet\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.660252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.657933 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-log-socket\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.660252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.657998 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-hostroot\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.660651 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.658445 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-cni-netd\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.660651 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.658510 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.660651 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.658613 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-socket-dir\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.660651 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.658617 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 18:53:37.660651 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.658658 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-etc-openvswitch\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.660651 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.659087 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/348e3292-41e0-44d6-b59b-81372e7d0b81-ovnkube-script-lib\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.660651 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:37.659163 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:37.660651 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:37.659221 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs podName:c46a7770-1771-4f8c-b35b-33ed971625c4 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:38.15920171 +0000 UTC m=+2.984364745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs") pod "network-metrics-daemon-kfvdz" (UID: "c46a7770-1771-4f8c-b35b-33ed971625c4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:37.660651 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.659383 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-kubelet\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.660651 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.659425 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.660651 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.659498 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-multus-daemon-config\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.660651 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.659588 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-var-lib-openvswitch\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.660651 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.659605 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-sys-fs\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.660651 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.659648 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.661059 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.659691 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-etc-kubernetes\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.661059 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.659746 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-multus-socket-dir-parent\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.661059 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.659799 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-cnibin\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.661059 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.659839 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/89a666fa-9965-49a8-930b-7fcebff2033b-registration-dir\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.661059 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.659842 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/348e3292-41e0-44d6-b59b-81372e7d0b81-host-cni-bin\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.663933 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.663592 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/348e3292-41e0-44d6-b59b-81372e7d0b81-ovn-node-metrics-cert\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.668319 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.667234 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l8br\" (UniqueName: \"kubernetes.io/projected/c46a7770-1771-4f8c-b35b-33ed971625c4-kube-api-access-4l8br\") pod \"network-metrics-daemon-kfvdz\" (UID: \"c46a7770-1771-4f8c-b35b-33ed971625c4\") " pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:37.668319 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.667689 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rpk6\" (UniqueName: \"kubernetes.io/projected/89a666fa-9965-49a8-930b-7fcebff2033b-kube-api-access-2rpk6\") pod \"aws-ebs-csi-driver-node-62nbt\" (UID: \"89a666fa-9965-49a8-930b-7fcebff2033b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.668463 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:37.668328 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:53:37.668463 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:37.668348 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:53:37.668463 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:37.668361 2569 projected.go:196] Error preparing data for projected volume kube-api-access-2vsgt for pod openshift-network-diagnostics/network-check-target-4hsrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:37.668463 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:37.668435 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt podName:3f612d36-6ce3-44a2-9a90-61dd316a6703 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:38.168411517 +0000 UTC m=+2.993574533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2vsgt" (UniqueName: "kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt") pod "network-check-target-4hsrp" (UID: "3f612d36-6ce3-44a2-9a90-61dd316a6703") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:37.668653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.668518 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2znm\" (UniqueName: \"kubernetes.io/projected/ab56b93f-4a0a-4f46-9910-94df2f70621f-kube-api-access-b2znm\") pod \"node-ca-bnbtq\" (UID: \"ab56b93f-4a0a-4f46-9910-94df2f70621f\") " pod="openshift-image-registry/node-ca-bnbtq" Jul 09 18:53:37.669885 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.669044 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz65d\" (UniqueName: \"kubernetes.io/projected/dfcb4a64-02f6-451c-97c0-451804e2bd36-kube-api-access-xz65d\") pod \"node-resolver-v6rsk\" (UID: \"dfcb4a64-02f6-451c-97c0-451804e2bd36\") " pod="openshift-dns/node-resolver-v6rsk" Jul 09 18:53:37.669885 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.669303 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w6fj\" (UniqueName: \"kubernetes.io/projected/a1e300ac-d943-40a5-847c-f33b16c01ab8-kube-api-access-6w6fj\") pod \"iptables-alerter-gmh5g\" (UID: \"a1e300ac-d943-40a5-847c-f33b16c01ab8\") " pod="openshift-network-operator/iptables-alerter-gmh5g" Jul 09 18:53:37.671174 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.670607 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd7d71e8-6272-469f-8569-7d97ac86c975-agent-certs\") pod \"konnectivity-agent-lmbqh\" (UID: \"dd7d71e8-6272-469f-8569-7d97ac86c975\") " pod="kube-system/konnectivity-agent-lmbqh" Jul 09 18:53:37.671174 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.671020 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n52n\" (UniqueName: \"kubernetes.io/projected/2472e3bc-2d37-4cb0-a8e6-5d8796e478e6-kube-api-access-9n52n\") pod \"multus-xxb68\" (UID: \"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6\") " pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.674095 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.674072 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qljjw\" (UniqueName: \"kubernetes.io/projected/348e3292-41e0-44d6-b59b-81372e7d0b81-kube-api-access-qljjw\") pod \"ovnkube-node-gsr6w\" (UID: \"348e3292-41e0-44d6-b59b-81372e7d0b81\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.693508 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.693437 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal" event={"ID":"dcabe690d34269c2f19f5d969bbfccfb","Type":"ContainerStarted","Data":"07e83c147c57f89856b1ece5c08d19bf04180758f1787187a676542155dfad6a"} Jul 09 18:53:37.694697 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.694672 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-185.ec2.internal" event={"ID":"58be071d26cafa55b0bf4c584a43070e","Type":"ContainerStarted","Data":"b07e5fb659e23458b1633630d0682d4e9923d6ea5170ee54aedbb93cf03098a5"} Jul 09 18:53:37.756702 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.756667 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ddf405a2-ba6c-4760-85e1-de6d4b699226-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.756856 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.756722 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:37.756856 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.756750 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-systemd\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.756856 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.756772 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-tmp\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.756856 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.756798 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddf405a2-ba6c-4760-85e1-de6d4b699226-cnibin\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.756856 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.756826 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddf405a2-ba6c-4760-85e1-de6d4b699226-system-cni-dir\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.757086 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.756879 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddf405a2-ba6c-4760-85e1-de6d4b699226-system-cni-dir\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.757086 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.756935 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-systemd\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.758908 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757411 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ddf405a2-ba6c-4760-85e1-de6d4b699226-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.758908 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757492 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-kubernetes\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.758908 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757521 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-948cm\" (UniqueName: \"kubernetes.io/projected/ddf405a2-ba6c-4760-85e1-de6d4b699226-kube-api-access-948cm\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.758908 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757544 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-sysconfig\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.758908 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757571 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-modprobe-d\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.758908 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757594 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-lib-modules\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.758908 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757626 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ddf405a2-ba6c-4760-85e1-de6d4b699226-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.758908 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757650 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-dbus\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:37.758908 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757672 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-sysctl-d\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.758908 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757694 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-tuned\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.758908 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757720 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-run\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.758908 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757743 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-host\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.758908 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757785 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-var-lib-kubelet\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.758908 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757801 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddf405a2-ba6c-4760-85e1-de6d4b699226-cnibin\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.759673 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757809 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zdfv\" (UniqueName: \"kubernetes.io/projected/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-kube-api-access-2zdfv\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.759673 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757877 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-kubelet-config\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:37.759673 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757930 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddf405a2-ba6c-4760-85e1-de6d4b699226-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.759673 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.757973 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-sysctl-conf\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.759673 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758053 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-kubelet-config\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:37.759673 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758099 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-sysctl-conf\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.759673 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758100 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ddf405a2-ba6c-4760-85e1-de6d4b699226-os-release\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.759673 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758129 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddf405a2-ba6c-4760-85e1-de6d4b699226-cni-binary-copy\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.759673 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758152 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-sys\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.759673 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758210 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddf405a2-ba6c-4760-85e1-de6d4b699226-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.759673 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758229 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-sys\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.759673 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758244 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-sysconfig\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.759673 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758274 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-kubernetes\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.759673 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:37.758305 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:37.760338 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:37.758364 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret podName:8de73e1e-1dc7-4c83-98fd-ac2697ca4c15 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:38.258347232 +0000 UTC m=+3.083510252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret") pod "global-pull-secret-syncer-kss86" (UID: "8de73e1e-1dc7-4c83-98fd-ac2697ca4c15") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:37.760338 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758415 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ddf405a2-ba6c-4760-85e1-de6d4b699226-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.760338 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758423 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ddf405a2-ba6c-4760-85e1-de6d4b699226-os-release\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.760338 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758552 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-modprobe-d\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.760338 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758663 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-lib-modules\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.760338 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758722 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-dbus\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:37.760338 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758771 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-host\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.760338 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758818 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-run\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.760338 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758867 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-var-lib-kubelet\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.760338 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.758868 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddf405a2-ba6c-4760-85e1-de6d4b699226-cni-binary-copy\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.760338 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.759006 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-sysctl-d\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.760862 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.760504 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-tmp\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.765023 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.764968 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-etc-tuned\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.775625 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.775599 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:53:37.780826 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.780803 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zdfv\" (UniqueName: \"kubernetes.io/projected/e35ca412-a2b2-4b4c-b8dd-de5286f3da01-kube-api-access-2zdfv\") pod \"tuned-s4jwv\" (UID: \"e35ca412-a2b2-4b4c-b8dd-de5286f3da01\") " pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.782848 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.782825 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-948cm\" (UniqueName: \"kubernetes.io/projected/ddf405a2-ba6c-4760-85e1-de6d4b699226-kube-api-access-948cm\") pod \"multus-additional-cni-plugins-9q928\" (UID: \"ddf405a2-ba6c-4760-85e1-de6d4b699226\") " pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:37.846075 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.846046 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" Jul 09 18:53:37.854130 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.854100 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lmbqh" Jul 09 18:53:37.857089 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:37.857062 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a666fa_9965_49a8_930b_7fcebff2033b.slice/crio-12b289f866ec37cdded2791f1315b7f42532b7f51c15ba72b77f92f02d9b4c5e WatchSource:0}: Error finding container 12b289f866ec37cdded2791f1315b7f42532b7f51c15ba72b77f92f02d9b4c5e: Status 404 returned error can't find the container with id 12b289f866ec37cdded2791f1315b7f42532b7f51c15ba72b77f92f02d9b4c5e Jul 09 18:53:37.862572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.862549 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gmh5g" Jul 09 18:53:37.862906 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:37.862880 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7d71e8_6272_469f_8569_7d97ac86c975.slice/crio-914a7837cf3a44190d6570e332e7274085919d74b680c616410d0032d222fec3 WatchSource:0}: Error finding container 914a7837cf3a44190d6570e332e7274085919d74b680c616410d0032d222fec3: Status 404 returned error can't find the container with id 914a7837cf3a44190d6570e332e7274085919d74b680c616410d0032d222fec3 Jul 09 18:53:37.871466 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:37.871440 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e300ac_d943_40a5_847c_f33b16c01ab8.slice/crio-c1e5d46580599f6f3f00db6e29f50d18742d7d69fc7de7eb5cecfe90c5bda160 WatchSource:0}: Error finding container c1e5d46580599f6f3f00db6e29f50d18742d7d69fc7de7eb5cecfe90c5bda160: Status 404 returned error can't find the container with id c1e5d46580599f6f3f00db6e29f50d18742d7d69fc7de7eb5cecfe90c5bda160 Jul 09 18:53:37.872612 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.872514 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v6rsk" Jul 09 18:53:37.879778 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.879572 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:53:37.888602 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.888562 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bnbtq" Jul 09 18:53:37.896118 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.896093 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xxb68" Jul 09 18:53:37.904855 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.904802 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" Jul 09 18:53:37.912720 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:37.912696 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9q928" Jul 09 18:53:38.160882 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:38.160798 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs\") pod \"network-metrics-daemon-kfvdz\" (UID: \"c46a7770-1771-4f8c-b35b-33ed971625c4\") " pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:38.161041 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:38.160921 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:38.161041 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:38.161000 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs podName:c46a7770-1771-4f8c-b35b-33ed971625c4 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:39.160981452 +0000 UTC m=+3.986144467 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs") pod "network-metrics-daemon-kfvdz" (UID: "c46a7770-1771-4f8c-b35b-33ed971625c4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:38.261174 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:38.261141 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsgt\" (UniqueName: \"kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt\") pod \"network-check-target-4hsrp\" (UID: \"3f612d36-6ce3-44a2-9a90-61dd316a6703\") " pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:38.261338 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:38.261194 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:38.261338 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:38.261296 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:38.261338 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:38.261313 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:53:38.261338 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:38.261335 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:53:38.261526 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:38.261344 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret podName:8de73e1e-1dc7-4c83-98fd-ac2697ca4c15 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:39.261331308 +0000 UTC m=+4.086494323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret") pod "global-pull-secret-syncer-kss86" (UID: "8de73e1e-1dc7-4c83-98fd-ac2697ca4c15") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:38.261526 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:38.261347 2569 projected.go:196] Error preparing data for projected volume kube-api-access-2vsgt for pod openshift-network-diagnostics/network-check-target-4hsrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:38.261526 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:38.261398 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt podName:3f612d36-6ce3-44a2-9a90-61dd316a6703 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:39.261380506 +0000 UTC m=+4.086543521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2vsgt" (UniqueName: "kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt") pod "network-check-target-4hsrp" (UID: "3f612d36-6ce3-44a2-9a90-61dd316a6703") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:38.603066 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:38.603022 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 18:48:37 +0000 UTC" deadline="2027-12-24 02:45:14.525385534 +0000 UTC" Jul 09 18:53:38.603066 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:38.603065 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12775h51m35.922325931s" Jul 09 18:53:38.688721 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:38.688691 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:38.689379 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:38.688697 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:38.689379 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:38.688803 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:53:38.689379 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:38.688889 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:53:38.698115 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:38.698086 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gmh5g" event={"ID":"a1e300ac-d943-40a5-847c-f33b16c01ab8","Type":"ContainerStarted","Data":"c1e5d46580599f6f3f00db6e29f50d18742d7d69fc7de7eb5cecfe90c5bda160"} Jul 09 18:53:38.699332 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:38.699306 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lmbqh" event={"ID":"dd7d71e8-6272-469f-8569-7d97ac86c975","Type":"ContainerStarted","Data":"914a7837cf3a44190d6570e332e7274085919d74b680c616410d0032d222fec3"} Jul 09 18:53:38.700631 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:38.700611 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" event={"ID":"89a666fa-9965-49a8-930b-7fcebff2033b","Type":"ContainerStarted","Data":"12b289f866ec37cdded2791f1315b7f42532b7f51c15ba72b77f92f02d9b4c5e"} Jul 09 18:53:38.950709 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:38.950627 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcb4a64_02f6_451c_97c0_451804e2bd36.slice/crio-efd7223361340819535d7e47ec9c64bc5c870503ea19a7e124452a5405295356 WatchSource:0}: Error finding container efd7223361340819535d7e47ec9c64bc5c870503ea19a7e124452a5405295356: Status 404 returned error can't find the container with id efd7223361340819535d7e47ec9c64bc5c870503ea19a7e124452a5405295356 Jul 09 18:53:38.987246 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:38.987214 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf405a2_ba6c_4760_85e1_de6d4b699226.slice/crio-8096b66ccb29e54e9678b9709bd8f289a77381dff5a71aaf391cad86fb8cfa3f WatchSource:0}: Error finding container 8096b66ccb29e54e9678b9709bd8f289a77381dff5a71aaf391cad86fb8cfa3f: Status 404 returned error can't find the container with id 8096b66ccb29e54e9678b9709bd8f289a77381dff5a71aaf391cad86fb8cfa3f Jul 09 18:53:38.988806 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:38.988775 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2472e3bc_2d37_4cb0_a8e6_5d8796e478e6.slice/crio-e0b00fcca0144898c7e0ddf64c4a3d4345e16e935251c42069dadb62f7d8c04e WatchSource:0}: Error finding container e0b00fcca0144898c7e0ddf64c4a3d4345e16e935251c42069dadb62f7d8c04e: Status 404 returned error can't find the container with id e0b00fcca0144898c7e0ddf64c4a3d4345e16e935251c42069dadb62f7d8c04e Jul 09 18:53:38.989855 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:38.989831 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348e3292_41e0_44d6_b59b_81372e7d0b81.slice/crio-7b398223c1dff1de8719185cd2dd09c8d94b00ef0366350700be35f1cc368c7c WatchSource:0}: Error finding container 7b398223c1dff1de8719185cd2dd09c8d94b00ef0366350700be35f1cc368c7c: Status 404 returned error can't find the container with id 7b398223c1dff1de8719185cd2dd09c8d94b00ef0366350700be35f1cc368c7c Jul 09 18:53:38.990867 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:38.990739 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab56b93f_4a0a_4f46_9910_94df2f70621f.slice/crio-3633d977bdce540d3630b60f28f4fc1d1b63bdb2bdd9bf3fafb21760d1a4b44f WatchSource:0}: Error finding container 3633d977bdce540d3630b60f28f4fc1d1b63bdb2bdd9bf3fafb21760d1a4b44f: Status 404 returned error can't find the container with id 3633d977bdce540d3630b60f28f4fc1d1b63bdb2bdd9bf3fafb21760d1a4b44f Jul 09 18:53:38.993154 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:53:38.993050 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35ca412_a2b2_4b4c_b8dd_de5286f3da01.slice/crio-cc40d8eeb89b9762bd179f6bc67c0a8e4a94cbaca9b83ebc83ed0d78f9937abb WatchSource:0}: Error finding container cc40d8eeb89b9762bd179f6bc67c0a8e4a94cbaca9b83ebc83ed0d78f9937abb: Status 404 returned error can't find the container with id cc40d8eeb89b9762bd179f6bc67c0a8e4a94cbaca9b83ebc83ed0d78f9937abb Jul 09 18:53:39.167488 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.167292 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs\") pod \"network-metrics-daemon-kfvdz\" (UID: \"c46a7770-1771-4f8c-b35b-33ed971625c4\") " pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:39.167635 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:39.167457 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:39.167635 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:39.167612 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs podName:c46a7770-1771-4f8c-b35b-33ed971625c4 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:41.167591803 +0000 UTC m=+5.992754821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs") pod "network-metrics-daemon-kfvdz" (UID: "c46a7770-1771-4f8c-b35b-33ed971625c4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:39.268597 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.268559 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsgt\" (UniqueName: \"kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt\") pod \"network-check-target-4hsrp\" (UID: \"3f612d36-6ce3-44a2-9a90-61dd316a6703\") " pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:39.268746 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.268628 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:39.268815 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:39.268806 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:39.268905 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:39.268893 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret podName:8de73e1e-1dc7-4c83-98fd-ac2697ca4c15 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:41.268874392 +0000 UTC m=+6.094037413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret") pod "global-pull-secret-syncer-kss86" (UID: "8de73e1e-1dc7-4c83-98fd-ac2697ca4c15") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:39.269026 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:39.269011 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:53:39.269088 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:39.269035 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:53:39.269088 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:39.269050 2569 projected.go:196] Error preparing data for projected volume kube-api-access-2vsgt for pod openshift-network-diagnostics/network-check-target-4hsrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:39.269088 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:39.269088 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt podName:3f612d36-6ce3-44a2-9a90-61dd316a6703 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:41.26907415 +0000 UTC m=+6.094237169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2vsgt" (UniqueName: "kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt") pod "network-check-target-4hsrp" (UID: "3f612d36-6ce3-44a2-9a90-61dd316a6703") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:39.604284 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.604184 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 18:48:37 +0000 UTC" deadline="2028-03-10 11:59:10.100588807 +0000 UTC" Jul 09 18:53:39.604284 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.604242 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14633h5m30.496350289s" Jul 09 18:53:39.691548 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.691520 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:39.692116 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:39.691641 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:53:39.710498 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.710258 2569 generic.go:356] "Generic (PLEG): container finished" podID="dcabe690d34269c2f19f5d969bbfccfb" containerID="7d7f5eebccc6b09ec96598d291cc3063f18b1c44c1b8584faaeddcb585682cc4" exitCode=0 Jul 09 18:53:39.714108 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.711223 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal" event={"ID":"dcabe690d34269c2f19f5d969bbfccfb","Type":"ContainerDied","Data":"7d7f5eebccc6b09ec96598d291cc3063f18b1c44c1b8584faaeddcb585682cc4"} Jul 09 18:53:39.714108 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.713697 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-185.ec2.internal" event={"ID":"58be071d26cafa55b0bf4c584a43070e","Type":"ContainerStarted","Data":"fa707e4845c21994643a96cba8868f685ea4e6acb11196410983dba37ba8d1a9"} Jul 09 18:53:39.716897 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.716837 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bnbtq" event={"ID":"ab56b93f-4a0a-4f46-9910-94df2f70621f","Type":"ContainerStarted","Data":"3633d977bdce540d3630b60f28f4fc1d1b63bdb2bdd9bf3fafb21760d1a4b44f"} Jul 09 18:53:39.719108 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.719057 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xxb68" event={"ID":"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6","Type":"ContainerStarted","Data":"e0b00fcca0144898c7e0ddf64c4a3d4345e16e935251c42069dadb62f7d8c04e"} Jul 09 18:53:39.722301 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.722276 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v6rsk" event={"ID":"dfcb4a64-02f6-451c-97c0-451804e2bd36","Type":"ContainerStarted","Data":"efd7223361340819535d7e47ec9c64bc5c870503ea19a7e124452a5405295356"} Jul 09 18:53:39.727935 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.727879 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" event={"ID":"e35ca412-a2b2-4b4c-b8dd-de5286f3da01","Type":"ContainerStarted","Data":"cc40d8eeb89b9762bd179f6bc67c0a8e4a94cbaca9b83ebc83ed0d78f9937abb"} Jul 09 18:53:39.731185 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.731079 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" event={"ID":"348e3292-41e0-44d6-b59b-81372e7d0b81","Type":"ContainerStarted","Data":"7b398223c1dff1de8719185cd2dd09c8d94b00ef0366350700be35f1cc368c7c"} Jul 09 18:53:39.737198 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.737153 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9q928" event={"ID":"ddf405a2-ba6c-4760-85e1-de6d4b699226","Type":"ContainerStarted","Data":"8096b66ccb29e54e9678b9709bd8f289a77381dff5a71aaf391cad86fb8cfa3f"} Jul 09 18:53:39.740019 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:39.739971 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-185.ec2.internal" podStartSLOduration=2.739936115 podStartE2EDuration="2.739936115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:39.739203545 +0000 UTC m=+4.564366581" watchObservedRunningTime="2026-07-09 18:53:39.739936115 +0000 UTC m=+4.565099153" Jul 09 18:53:40.689812 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:40.689280 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:40.689812 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:40.689407 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:53:40.689812 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:40.689713 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:40.689812 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:40.689774 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:53:41.186347 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:41.185767 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs\") pod \"network-metrics-daemon-kfvdz\" (UID: \"c46a7770-1771-4f8c-b35b-33ed971625c4\") " pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:41.186347 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:41.185942 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:41.186347 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:41.186022 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs podName:c46a7770-1771-4f8c-b35b-33ed971625c4 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:45.1860012 +0000 UTC m=+10.011164222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs") pod "network-metrics-daemon-kfvdz" (UID: "c46a7770-1771-4f8c-b35b-33ed971625c4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:41.286871 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:41.286824 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsgt\" (UniqueName: \"kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt\") pod \"network-check-target-4hsrp\" (UID: \"3f612d36-6ce3-44a2-9a90-61dd316a6703\") " pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:41.287046 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:41.286904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:41.287108 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:41.287074 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:41.287157 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:41.287138 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret podName:8de73e1e-1dc7-4c83-98fd-ac2697ca4c15 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:45.287119577 +0000 UTC m=+10.112282598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret") pod "global-pull-secret-syncer-kss86" (UID: "8de73e1e-1dc7-4c83-98fd-ac2697ca4c15") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:41.287564 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:41.287543 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:53:41.287664 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:41.287574 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:53:41.287664 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:41.287590 2569 projected.go:196] Error preparing data for projected volume kube-api-access-2vsgt for pod openshift-network-diagnostics/network-check-target-4hsrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:41.287664 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:41.287638 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt podName:3f612d36-6ce3-44a2-9a90-61dd316a6703 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:45.287622294 +0000 UTC m=+10.112785323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2vsgt" (UniqueName: "kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt") pod "network-check-target-4hsrp" (UID: "3f612d36-6ce3-44a2-9a90-61dd316a6703") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:41.688353 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:41.688320 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:41.688519 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:41.688465 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:53:42.688779 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:42.688734 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:42.689262 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:42.688879 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:53:42.689328 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:42.689304 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:42.689429 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:42.689406 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:53:43.688939 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:43.688856 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:43.689347 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:43.689007 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:53:44.689009 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:44.688974 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:44.689420 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:44.688974 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:44.689420 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:44.689120 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:53:44.689420 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:44.689161 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:53:45.218703 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:45.218665 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs\") pod \"network-metrics-daemon-kfvdz\" (UID: \"c46a7770-1771-4f8c-b35b-33ed971625c4\") " pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:45.218889 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:45.218833 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:45.218971 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:45.218893 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs podName:c46a7770-1771-4f8c-b35b-33ed971625c4 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:53.218874342 +0000 UTC m=+18.044037364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs") pod "network-metrics-daemon-kfvdz" (UID: "c46a7770-1771-4f8c-b35b-33ed971625c4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:45.319487 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:45.319453 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:45.319676 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:45.319536 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsgt\" (UniqueName: \"kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt\") pod \"network-check-target-4hsrp\" (UID: \"3f612d36-6ce3-44a2-9a90-61dd316a6703\") " pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:45.319676 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:45.319630 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:45.319676 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:45.319665 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:53:45.319836 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:45.319683 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:53:45.319836 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:45.319695 2569 projected.go:196] Error preparing data for projected volume kube-api-access-2vsgt for pod openshift-network-diagnostics/network-check-target-4hsrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:45.319836 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:45.319702 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret podName:8de73e1e-1dc7-4c83-98fd-ac2697ca4c15 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:53.319680405 +0000 UTC m=+18.144843422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret") pod "global-pull-secret-syncer-kss86" (UID: "8de73e1e-1dc7-4c83-98fd-ac2697ca4c15") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:45.319836 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:45.319737 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt podName:3f612d36-6ce3-44a2-9a90-61dd316a6703 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:53.319720694 +0000 UTC m=+18.144883709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2vsgt" (UniqueName: "kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt") pod "network-check-target-4hsrp" (UID: "3f612d36-6ce3-44a2-9a90-61dd316a6703") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:45.692046 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:45.692015 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:45.692505 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:45.692123 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:53:46.688452 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:46.688416 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:46.688452 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:46.688463 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:46.688666 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:46.688551 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:53:46.688738 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:46.688706 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:53:47.694438 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:47.694406 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:47.694803 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:47.694540 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:53:47.756682 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:47.756645 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" event={"ID":"89a666fa-9965-49a8-930b-7fcebff2033b","Type":"ContainerStarted","Data":"aaeaf1f5d28381110d19872640b9f6c9dddef19fa4a3791451194fe33d6bcf64"} Jul 09 18:53:47.758564 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:47.758531 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" event={"ID":"e35ca412-a2b2-4b4c-b8dd-de5286f3da01","Type":"ContainerStarted","Data":"f2477ab9560ae2ae179e043fd7199d20416466060a91c000880be59fff59601f"} Jul 09 18:53:47.760699 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:47.760674 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9q928" event={"ID":"ddf405a2-ba6c-4760-85e1-de6d4b699226","Type":"ContainerStarted","Data":"101031849ba337f11edd1db774af8181b3664e6d6e276f68c2d6ee1fdb8a6641"} Jul 09 18:53:47.763014 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:47.762977 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal" event={"ID":"dcabe690d34269c2f19f5d969bbfccfb","Type":"ContainerStarted","Data":"1970412066cb70b5dec330e2e497790ecf5dc6a6484663d0cf9dc35b8731fe34"} Jul 09 18:53:47.765094 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:47.765059 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bnbtq" event={"ID":"ab56b93f-4a0a-4f46-9910-94df2f70621f","Type":"ContainerStarted","Data":"91465c774606168728bba45290cfd83136a2ecbbac5ad7aee3797cf5fb09e73c"} Jul 09 18:53:47.767602 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:47.767324 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v6rsk" event={"ID":"dfcb4a64-02f6-451c-97c0-451804e2bd36","Type":"ContainerStarted","Data":"9e8bc834eeb2dd98422ab740526b1a44461c6b5bf3eedbd666d14a364e2d2557"} Jul 09 18:53:47.769113 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:47.769089 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lmbqh" event={"ID":"dd7d71e8-6272-469f-8569-7d97ac86c975","Type":"ContainerStarted","Data":"1048fa6458b24d19c9aab40515d2061dceff404f69137021310b61ae53416c93"} Jul 09 18:53:47.800576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:47.800521 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s4jwv" podStartSLOduration=4.38763357 podStartE2EDuration="12.800501989s" totalImagesPullingTime="8.412868419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:47.779204978 +0000 UTC m=+12.604368009" watchObservedRunningTime="2026-07-09 18:53:47.800501989 +0000 UTC m=+12.625665026" Jul 09 18:53:47.801104 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:47.801063 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v6rsk" podStartSLOduration=4.392997139 podStartE2EDuration="12.801053054s" totalImagesPullingTime="8.408055915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:47.800741949 +0000 UTC m=+12.625904987" watchObservedRunningTime="2026-07-09 18:53:47.801053054 +0000 UTC m=+12.626216091" Jul 09 18:53:47.870214 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:47.869943 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lmbqh" podStartSLOduration=3.363287985 podStartE2EDuration="12.869928402s" totalImagesPullingTime="9.506640417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:47.869812078 +0000 UTC m=+12.694975116" watchObservedRunningTime="2026-07-09 18:53:47.869928402 +0000 UTC m=+12.695091439" Jul 09 18:53:47.870641 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:47.870589 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-185.ec2.internal" podStartSLOduration=10.870578088 podStartE2EDuration="10.870578088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:47.852016279 +0000 UTC m=+12.677179327" watchObservedRunningTime="2026-07-09 18:53:47.870578088 +0000 UTC m=+12.695741126" Jul 09 18:53:47.887707 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:47.887664 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bnbtq" podStartSLOduration=4.508813759 podStartE2EDuration="12.88764705s" totalImagesPullingTime="8.378833291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:47.887207784 +0000 UTC m=+12.712370834" watchObservedRunningTime="2026-07-09 18:53:47.88764705 +0000 UTC m=+12.712810089" Jul 09 18:53:48.688774 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:48.688741 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:48.688942 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:48.688864 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:53:48.688942 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:48.688894 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:48.689085 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:48.688965 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:53:48.774851 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:48.774817 2569 generic.go:356] "Generic (PLEG): container finished" podID="ddf405a2-ba6c-4760-85e1-de6d4b699226" containerID="101031849ba337f11edd1db774af8181b3664e6d6e276f68c2d6ee1fdb8a6641" exitCode=0 Jul 09 18:53:48.775250 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:48.774922 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9q928" event={"ID":"ddf405a2-ba6c-4760-85e1-de6d4b699226","Type":"ContainerDied","Data":"101031849ba337f11edd1db774af8181b3664e6d6e276f68c2d6ee1fdb8a6641"} Jul 09 18:53:49.064424 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:49.064403 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 18:53:49.597334 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:49.597219 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:53:49.064420476Z","UUID":"c3628894-061c-4809-90d5-d18198eac233","Handler":null,"Name":"","Endpoint":""} Jul 09 18:53:49.599150 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:49.599129 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 18:53:49.599278 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:49.599171 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 18:53:49.690432 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:49.688940 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:49.690432 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:49.690094 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:53:49.778940 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:49.778908 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gmh5g" event={"ID":"a1e300ac-d943-40a5-847c-f33b16c01ab8","Type":"ContainerStarted","Data":"e04a495c240efcd9d99c40bb1054253db41c2aa86e88abfee6872d55646ce1f3"} Jul 09 18:53:49.780852 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:49.780804 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" event={"ID":"89a666fa-9965-49a8-930b-7fcebff2033b","Type":"ContainerStarted","Data":"65ae6e03083c8780cc7d2b8bd66404ea359d38aa5502621d1cd608829311a50e"} Jul 09 18:53:49.798297 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:49.797135 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gmh5g" podStartSLOduration=5.293944421 podStartE2EDuration="14.797108034s" totalImagesPullingTime="9.503163613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:49.795070137 +0000 UTC m=+14.620233176" watchObservedRunningTime="2026-07-09 18:53:49.797108034 +0000 UTC m=+14.622271084" Jul 09 18:53:50.688348 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:50.688320 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:50.688456 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:50.688337 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:50.688550 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:50.688454 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:53:50.688550 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:50.688478 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:53:50.788609 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:50.788571 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" event={"ID":"89a666fa-9965-49a8-930b-7fcebff2033b","Type":"ContainerStarted","Data":"3adbc91a7138d6e72c1ca9ba8909a1428a7a976977b06a1cddc7f1140ca4bdc1"} Jul 09 18:53:51.689182 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:51.689147 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:51.689350 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:51.689275 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:53:52.467601 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:52.467567 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lmbqh" Jul 09 18:53:52.468462 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:52.468441 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lmbqh" Jul 09 18:53:52.489414 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:52.489360 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62nbt" podStartSLOduration=5.1887757820000004 podStartE2EDuration="17.489338066s" totalImagesPullingTime="12.300562284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:50.812403666 +0000 UTC m=+15.637566703" watchObservedRunningTime="2026-07-09 18:53:52.489338066 +0000 UTC m=+17.314501104" Jul 09 18:53:52.689105 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:52.689072 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:52.689105 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:52.689102 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:52.689343 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:52.689225 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:53:52.689402 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:52.689375 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:53:53.290093 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:53.290056 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs\") pod \"network-metrics-daemon-kfvdz\" (UID: \"c46a7770-1771-4f8c-b35b-33ed971625c4\") " pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:53.290252 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:53.290204 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:53.290291 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:53.290264 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs podName:c46a7770-1771-4f8c-b35b-33ed971625c4 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:09.290247364 +0000 UTC m=+34.115410381 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs") pod "network-metrics-daemon-kfvdz" (UID: "c46a7770-1771-4f8c-b35b-33ed971625c4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:53.390740 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:53.390539 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:53.390930 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:53.390688 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:53.390930 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:53.390793 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsgt\" (UniqueName: \"kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt\") pod \"network-check-target-4hsrp\" (UID: \"3f612d36-6ce3-44a2-9a90-61dd316a6703\") " pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:53.390930 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:53.390853 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret podName:8de73e1e-1dc7-4c83-98fd-ac2697ca4c15 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:09.390834084 +0000 UTC m=+34.215997103 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret") pod "global-pull-secret-syncer-kss86" (UID: "8de73e1e-1dc7-4c83-98fd-ac2697ca4c15") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:53.390930 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:53.390921 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:53:53.391156 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:53.390937 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:53:53.391156 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:53.390962 2569 projected.go:196] Error preparing data for projected volume kube-api-access-2vsgt for pod openshift-network-diagnostics/network-check-target-4hsrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:53.391156 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:53.391016 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt podName:3f612d36-6ce3-44a2-9a90-61dd316a6703 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:09.391001283 +0000 UTC m=+34.216164316 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2vsgt" (UniqueName: "kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt") pod "network-check-target-4hsrp" (UID: "3f612d36-6ce3-44a2-9a90-61dd316a6703") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:53.689244 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:53.689174 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:53.689632 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:53.689309 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:53:54.688633 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:54.688601 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:54.688802 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:54.688736 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:53:54.688802 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:54.688759 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:54.688973 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:54.688886 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:53:55.691187 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:55.691159 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:55.691659 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:55.691256 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:53:56.688708 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:56.688668 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:56.688892 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:56.688669 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:56.688892 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:56.688805 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:53:56.688892 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:56.688882 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:53:57.689105 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:57.689071 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:57.689682 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:57.689196 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:53:58.688765 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:58.688728 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:53:58.688968 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:58.688841 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:53:58.688968 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:58.688906 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:53:58.689090 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:58.689014 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:53:59.688574 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:53:59.688537 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:53:59.689201 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:53:59.688676 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:00.689152 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:00.689119 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:00.689570 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:00.689221 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:00.689570 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:00.689284 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:00.689570 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:00.689396 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:01.069226 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:01.069202 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v6rsk_dfcb4a64-02f6-451c-97c0-451804e2bd36/dns-node-resolver/0.log" Jul 09 18:54:01.689260 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:01.689028 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:01.689686 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:01.689292 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:01.810832 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:01.810794 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xxb68" event={"ID":"2472e3bc-2d37-4cb0-a8e6-5d8796e478e6","Type":"ContainerStarted","Data":"0cd2361f7a68b418d76ee204a6241a9bb6f8bac8c33baa2ac3687d788c16f5a1"} Jul 09 18:54:01.813459 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:01.813438 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 18:54:01.813788 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:01.813762 2569 generic.go:356] "Generic (PLEG): container finished" podID="348e3292-41e0-44d6-b59b-81372e7d0b81" containerID="8064e639f5f2ae9c54e61be75e5c79d4f350362d0fa3fb6e38be2640902a4931" exitCode=1 Jul 09 18:54:01.813864 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:01.813835 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" event={"ID":"348e3292-41e0-44d6-b59b-81372e7d0b81","Type":"ContainerStarted","Data":"033dbbdcbaf83748577a092f42f21afbf30155ec52995ba1fcbecc215fd9446c"} Jul 09 18:54:01.813913 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:01.813892 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" event={"ID":"348e3292-41e0-44d6-b59b-81372e7d0b81","Type":"ContainerStarted","Data":"7acc08901213684689a3832ad9e69403807548d012175403c45d7be5906d0f9f"} Jul 09 18:54:01.813992 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:01.813932 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" event={"ID":"348e3292-41e0-44d6-b59b-81372e7d0b81","Type":"ContainerStarted","Data":"82a35c9ce0cc2c5f5c39b132f25507bf8c528e3f16b94d23a7811179dd058afc"} Jul 09 18:54:01.813992 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:01.813983 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" event={"ID":"348e3292-41e0-44d6-b59b-81372e7d0b81","Type":"ContainerStarted","Data":"b32a63fd9cc6f3e431a6bfef7c963efc71a5efda99cfa7d296e94582460f864b"} Jul 09 18:54:01.814095 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:01.814017 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" event={"ID":"348e3292-41e0-44d6-b59b-81372e7d0b81","Type":"ContainerDied","Data":"8064e639f5f2ae9c54e61be75e5c79d4f350362d0fa3fb6e38be2640902a4931"} Jul 09 18:54:01.814095 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:01.814046 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" event={"ID":"348e3292-41e0-44d6-b59b-81372e7d0b81","Type":"ContainerStarted","Data":"00f3c71dc54d9b32eeedc09742802586acf62856c39f660b93516c891dc51b0e"} Jul 09 18:54:01.815553 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:01.815531 2569 generic.go:356] "Generic (PLEG): container finished" podID="ddf405a2-ba6c-4760-85e1-de6d4b699226" containerID="0c515fef9948fb4c8de12a4fc5ae2c06448dc0190411a446f8d8fb423059958f" exitCode=0 Jul 09 18:54:01.815637 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:01.815562 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9q928" event={"ID":"ddf405a2-ba6c-4760-85e1-de6d4b699226","Type":"ContainerDied","Data":"0c515fef9948fb4c8de12a4fc5ae2c06448dc0190411a446f8d8fb423059958f"} Jul 09 18:54:01.852758 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:01.852729 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bnbtq_ab56b93f-4a0a-4f46-9910-94df2f70621f/node-ca/0.log" Jul 09 18:54:01.890666 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:01.890619 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xxb68" podStartSLOduration=4.860484656 podStartE2EDuration="26.89059986s" totalImagesPullingTime="22.030115204s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:01.836618954 +0000 UTC m=+26.661781991" watchObservedRunningTime="2026-07-09 18:54:01.89059986 +0000 UTC m=+26.715762896" Jul 09 18:54:02.688598 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:02.688564 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:02.688765 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:02.688693 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:02.688765 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:02.688747 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:02.688889 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:02.688854 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:03.688906 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:03.688869 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:03.689411 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:03.689058 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:03.823799 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:03.823771 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 18:54:03.824195 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:03.824170 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" event={"ID":"348e3292-41e0-44d6-b59b-81372e7d0b81","Type":"ContainerStarted","Data":"38cdae029191f324b4968ff3b2c291b96caaab7d3b71e9d4590dfd70a182eeaa"} Jul 09 18:54:03.826165 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:03.826144 2569 generic.go:356] "Generic (PLEG): container finished" podID="ddf405a2-ba6c-4760-85e1-de6d4b699226" containerID="e16abfe86b277a3644cb4ab4e84fa54d52a06b4aaf30ac17f9612718a5a8d71a" exitCode=0 Jul 09 18:54:03.826238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:03.826189 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9q928" event={"ID":"ddf405a2-ba6c-4760-85e1-de6d4b699226","Type":"ContainerDied","Data":"e16abfe86b277a3644cb4ab4e84fa54d52a06b4aaf30ac17f9612718a5a8d71a"} Jul 09 18:54:04.688561 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:04.688537 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:04.688653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:04.688570 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:04.688653 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:04.688643 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:04.688760 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:04.688745 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:04.830331 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:04.830298 2569 generic.go:356] "Generic (PLEG): container finished" podID="ddf405a2-ba6c-4760-85e1-de6d4b699226" containerID="33b9a0ebb80a05687815c5a66863fd2379e363695d6d6029ddcaf3fa24db7f59" exitCode=0 Jul 09 18:54:04.830746 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:04.830356 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9q928" event={"ID":"ddf405a2-ba6c-4760-85e1-de6d4b699226","Type":"ContainerDied","Data":"33b9a0ebb80a05687815c5a66863fd2379e363695d6d6029ddcaf3fa24db7f59"} Jul 09 18:54:05.690324 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:05.690135 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:05.690477 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:05.690449 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:05.838569 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:05.838542 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 18:54:05.838998 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:05.838874 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" event={"ID":"348e3292-41e0-44d6-b59b-81372e7d0b81","Type":"ContainerStarted","Data":"59e9a7835b66e44b30a5db6b9ca9e73baf6151dc3cc0b1e4f73ffce62e654a01"} Jul 09 18:54:05.839181 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:05.839158 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:54:05.839410 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:05.839396 2569 scope.go:117] "RemoveContainer" containerID="8064e639f5f2ae9c54e61be75e5c79d4f350362d0fa3fb6e38be2640902a4931" Jul 09 18:54:05.857570 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:05.857544 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:54:06.688585 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:06.688551 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:06.688752 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:06.688601 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:06.688752 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:06.688669 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:06.688858 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:06.688757 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:06.844359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:06.844330 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 18:54:06.844721 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:06.844674 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" event={"ID":"348e3292-41e0-44d6-b59b-81372e7d0b81","Type":"ContainerStarted","Data":"1c2cde58afdd80ce135bae3738a60c26e23bdbcff973cb0ec847923757759ef3"} Jul 09 18:54:06.844962 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:06.844922 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:54:06.845067 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:06.845002 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:54:06.860811 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:06.860787 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:54:06.943615 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:06.943521 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" podStartSLOduration=9.945975654 podStartE2EDuration="31.943505673s" totalImagesPullingTime="21.997530019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:06.898462603 +0000 UTC m=+31.723625641" watchObservedRunningTime="2026-07-09 18:54:06.943505673 +0000 UTC m=+31.768668709" Jul 09 18:54:07.313304 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:07.313269 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lmbqh" Jul 09 18:54:07.313482 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:07.313419 2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 18:54:07.313836 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:07.313814 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lmbqh" Jul 09 18:54:07.688533 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:07.688459 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:07.688667 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:07.688562 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:08.688602 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:08.688567 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:08.689116 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:08.688694 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:08.689116 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:08.688750 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:08.689116 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:08.688837 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:09.306733 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:09.306650 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs\") pod \"network-metrics-daemon-kfvdz\" (UID: \"c46a7770-1771-4f8c-b35b-33ed971625c4\") " pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:09.306986 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:09.306811 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:54:09.306986 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:09.306871 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs podName:c46a7770-1771-4f8c-b35b-33ed971625c4 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:41.306856325 +0000 UTC m=+66.132019340 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs") pod "network-metrics-daemon-kfvdz" (UID: "c46a7770-1771-4f8c-b35b-33ed971625c4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:54:09.407970 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:09.407930 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:09.408148 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:09.408016 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsgt\" (UniqueName: \"kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt\") pod \"network-check-target-4hsrp\" (UID: \"3f612d36-6ce3-44a2-9a90-61dd316a6703\") " pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:09.408148 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:09.408106 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:54:09.408148 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:09.408128 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:54:09.408148 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:09.408146 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:54:09.408334 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:09.408158 2569 projected.go:196] Error preparing data for projected volume kube-api-access-2vsgt for pod openshift-network-diagnostics/network-check-target-4hsrp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:54:09.408334 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:09.408180 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret podName:8de73e1e-1dc7-4c83-98fd-ac2697ca4c15 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:41.408160997 +0000 UTC m=+66.233324030 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret") pod "global-pull-secret-syncer-kss86" (UID: "8de73e1e-1dc7-4c83-98fd-ac2697ca4c15") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:54:09.408334 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:09.408198 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt podName:3f612d36-6ce3-44a2-9a90-61dd316a6703 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:41.408187502 +0000 UTC m=+66.233350516 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2vsgt" (UniqueName: "kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt") pod "network-check-target-4hsrp" (UID: "3f612d36-6ce3-44a2-9a90-61dd316a6703") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:54:09.688984 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:09.688877 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:09.689436 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:09.689033 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:10.688485 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:10.688395 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:10.688625 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:10.688395 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:10.688625 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:10.688521 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:10.688625 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:10.688583 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:10.856031 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:10.856000 2569 generic.go:356] "Generic (PLEG): container finished" podID="ddf405a2-ba6c-4760-85e1-de6d4b699226" containerID="707b9d790695657eff93e8a22f39072c813e698c1f3b622193f3afd45d309db5" exitCode=0 Jul 09 18:54:10.856031 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:10.856034 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9q928" event={"ID":"ddf405a2-ba6c-4760-85e1-de6d4b699226","Type":"ContainerDied","Data":"707b9d790695657eff93e8a22f39072c813e698c1f3b622193f3afd45d309db5"} Jul 09 18:54:11.689187 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:11.689152 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:11.689358 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:11.689258 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:11.861721 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:11.861689 2569 generic.go:356] "Generic (PLEG): container finished" podID="ddf405a2-ba6c-4760-85e1-de6d4b699226" containerID="fb931e5ecb593a39d38230e4cc3daaba1f0e8d224dbd3fb7afaab903bcf112a0" exitCode=0 Jul 09 18:54:11.861721 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:11.861727 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9q928" event={"ID":"ddf405a2-ba6c-4760-85e1-de6d4b699226","Type":"ContainerDied","Data":"fb931e5ecb593a39d38230e4cc3daaba1f0e8d224dbd3fb7afaab903bcf112a0"} Jul 09 18:54:12.688459 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:12.688423 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:12.688606 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:12.688425 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:12.688606 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:12.688534 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:12.688759 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:12.688615 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:12.867414 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:12.867380 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9q928" event={"ID":"ddf405a2-ba6c-4760-85e1-de6d4b699226","Type":"ContainerStarted","Data":"592f2b7d5c0ab3876d3f1d28d514653aea2f9eeaa873c1226dff855f43987ae5"} Jul 09 18:54:13.689088 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:13.689051 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:13.689396 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:13.689360 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:14.689210 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:14.689176 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:14.689584 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:14.689281 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:14.689584 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:14.689336 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:14.689584 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:14.689418 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:15.689464 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:15.689436 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:15.689816 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:15.689516 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:16.688212 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:16.688175 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:16.688388 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:16.688175 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:16.688388 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:16.688291 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:16.688388 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:16.688369 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:17.688968 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:17.688910 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:17.689362 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:17.689040 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:18.688991 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:18.688943 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:18.689390 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:18.688941 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:18.689390 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:18.689065 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:18.689390 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:18.689129 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:19.688805 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:19.688769 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:19.688982 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:19.688899 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:20.688270 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:20.688233 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:20.688640 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:20.688348 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:20.688640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:20.688397 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:20.688640 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:20.688498 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:21.689051 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:21.689021 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:21.689421 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:21.689123 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:22.688650 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:22.688614 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:22.688650 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:22.688655 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:22.688894 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:22.688728 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:22.688894 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:22.688868 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:23.688704 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:23.688667 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:23.689123 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:23.688798 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:24.688917 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:24.688870 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:24.688917 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:24.688904 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:24.689351 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:24.689050 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:24.689351 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:24.689176 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:25.689373 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:25.689204 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:25.689734 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:25.689426 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:26.688836 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:26.688799 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:26.689042 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:26.688799 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:26.689042 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:26.688910 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:26.689042 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:26.689019 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:27.689251 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:27.689215 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:27.689639 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:27.689348 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:28.688514 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:28.688477 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:28.688514 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:28.688504 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:28.688730 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:28.688582 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:28.688730 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:28.688688 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:29.688557 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:29.688529 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:29.689023 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:29.688640 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:30.689194 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:30.689152 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:30.689731 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:30.689153 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:30.689731 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:30.689281 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:30.689731 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:30.689333 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:31.688893 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:31.688859 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:31.689104 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:31.688999 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:32.689261 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:32.689224 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:32.689261 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:32.689256 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:32.689685 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:32.689351 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:32.689685 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:32.689470 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:33.688916 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:33.688883 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:33.689117 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:33.689015 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:34.689013 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:34.688977 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:34.689013 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:34.689002 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:34.689441 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:34.689094 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:34.689441 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:34.689200 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:35.325745 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:35.325662 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9q928" podStartSLOduration=32.384916674 podStartE2EDuration="1m0.325643686s" totalImagesPullingTime="27.940727012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:12.897693723 +0000 UTC m=+37.722856759" watchObservedRunningTime="2026-07-09 18:54:35.325643686 +0000 UTC m=+60.150806724" Jul 09 18:54:35.326197 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:35.326180 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4hsrp"] Jul 09 18:54:35.326295 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:35.326278 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:35.326377 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:35.326354 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:35.329279 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:35.329253 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kss86"] Jul 09 18:54:35.329419 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:35.329355 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:35.329484 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:35.329455 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:35.330441 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:35.330102 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kfvdz"] Jul 09 18:54:35.330441 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:35.330195 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:35.330441 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:35.330295 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:36.688939 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:36.688894 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:36.689645 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:36.689011 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:36.689645 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:36.689022 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:36.689645 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:36.689019 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hsrp" podUID="3f612d36-6ce3-44a2-9a90-61dd316a6703" Jul 09 18:54:36.689645 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:36.689086 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kss86" podUID="8de73e1e-1dc7-4c83-98fd-ac2697ca4c15" Jul 09 18:54:36.689645 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:36.689184 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfvdz" podUID="c46a7770-1771-4f8c-b35b-33ed971625c4" Jul 09 18:54:38.509661 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.509634 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-185.ec2.internal" event="NodeReady" Jul 09 18:54:38.510032 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.509767 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 18:54:38.547744 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.547713 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dbb5989d9-7p7xp"] Jul 09 18:54:38.549895 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.549874 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.552359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.552331 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4w25g\"" Jul 09 18:54:38.552359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.552348 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 18:54:38.552359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.552357 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 18:54:38.552565 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.552373 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 18:54:38.558724 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.558706 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 18:54:38.560646 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.560626 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qxft6"] Jul 09 18:54:38.562236 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.562222 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qxft6" Jul 09 18:54:38.564645 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.564626 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 18:54:38.564759 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.564743 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 18:54:38.564836 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.564819 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 18:54:38.564913 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.564894 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tl7rk\"" Jul 09 18:54:38.569548 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.569527 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4wknv"] Jul 09 18:54:38.574181 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.574158 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dbb5989d9-7p7xp"] Jul 09 18:54:38.574688 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.574671 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.576284 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.576260 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qxft6"] Jul 09 18:54:38.577396 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.577377 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 18:54:38.577487 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.577416 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 18:54:38.577613 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.577598 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 18:54:38.577613 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.577609 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k2d2w\"" Jul 09 18:54:38.577711 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.577692 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 18:54:38.584905 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.584862 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4wknv"] Jul 09 18:54:38.620247 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.620209 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88d1f57b-538f-4ef6-b0e8-616c9c493dab-registry-certificates\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.620247 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.620248 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88d1f57b-538f-4ef6-b0e8-616c9c493dab-trusted-ca\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.620442 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.620266 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2tlm\" (UniqueName: \"kubernetes.io/projected/88d1f57b-538f-4ef6-b0e8-616c9c493dab-kube-api-access-b2tlm\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.620442 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.620323 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88d1f57b-538f-4ef6-b0e8-616c9c493dab-image-registry-private-configuration\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.620442 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.620369 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88d1f57b-538f-4ef6-b0e8-616c9c493dab-registry-tls\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.620442 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.620406 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bd52903-4c1f-4840-a92b-c1ff27ec76be-cert\") pod \"ingress-canary-qxft6\" (UID: \"3bd52903-4c1f-4840-a92b-c1ff27ec76be\") " pod="openshift-ingress-canary/ingress-canary-qxft6" Jul 09 18:54:38.620442 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.620432 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxgsn\" (UniqueName: \"kubernetes.io/projected/3bd52903-4c1f-4840-a92b-c1ff27ec76be-kube-api-access-pxgsn\") pod \"ingress-canary-qxft6\" (UID: \"3bd52903-4c1f-4840-a92b-c1ff27ec76be\") " pod="openshift-ingress-canary/ingress-canary-qxft6" Jul 09 18:54:38.620603 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.620452 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88d1f57b-538f-4ef6-b0e8-616c9c493dab-ca-trust-extracted\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.620603 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.620490 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88d1f57b-538f-4ef6-b0e8-616c9c493dab-bound-sa-token\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.620603 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.620511 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88d1f57b-538f-4ef6-b0e8-616c9c493dab-installation-pull-secrets\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.658890 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.658821 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lvppt"] Jul 09 18:54:38.660678 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.660663 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:38.662871 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.662848 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 18:54:38.662985 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.662879 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xvss5\"" Jul 09 18:54:38.662985 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.662888 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 18:54:38.674576 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.674554 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lvppt"] Jul 09 18:54:38.688990 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.688960 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:38.689097 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.689053 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:38.689097 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.689078 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:38.691485 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.691467 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 18:54:38.693098 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.693084 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 18:54:38.693185 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.693164 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47bt6\"" Jul 09 18:54:38.693416 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.693399 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 18:54:38.693416 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.693410 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qkpmb\"" Jul 09 18:54:38.693537 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.693425 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 18:54:38.721000 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.720974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88d1f57b-538f-4ef6-b0e8-616c9c493dab-registry-certificates\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.721000 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721007 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88d1f57b-538f-4ef6-b0e8-616c9c493dab-trusted-ca\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.721178 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2tlm\" (UniqueName: \"kubernetes.io/projected/88d1f57b-538f-4ef6-b0e8-616c9c493dab-kube-api-access-b2tlm\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.721178 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721058 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f2246db-5e70-45a8-94a5-4f387631fe03-metrics-tls\") pod \"dns-default-lvppt\" (UID: \"8f2246db-5e70-45a8-94a5-4f387631fe03\") " pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:38.721178 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721086 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9293675-16ef-4401-af79-aa74f9597e8f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wknv\" (UID: \"d9293675-16ef-4401-af79-aa74f9597e8f\") " pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.721178 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721124 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88d1f57b-538f-4ef6-b0e8-616c9c493dab-image-registry-private-configuration\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.721178 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721170 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88d1f57b-538f-4ef6-b0e8-616c9c493dab-registry-tls\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.721410 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721208 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9293675-16ef-4401-af79-aa74f9597e8f-data-volume\") pod \"insights-runtime-extractor-4wknv\" (UID: \"d9293675-16ef-4401-af79-aa74f9597e8f\") " pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.721410 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721277 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bd52903-4c1f-4840-a92b-c1ff27ec76be-cert\") pod \"ingress-canary-qxft6\" (UID: \"3bd52903-4c1f-4840-a92b-c1ff27ec76be\") " pod="openshift-ingress-canary/ingress-canary-qxft6" Jul 09 18:54:38.721410 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721323 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxgsn\" (UniqueName: \"kubernetes.io/projected/3bd52903-4c1f-4840-a92b-c1ff27ec76be-kube-api-access-pxgsn\") pod \"ingress-canary-qxft6\" (UID: \"3bd52903-4c1f-4840-a92b-c1ff27ec76be\") " pod="openshift-ingress-canary/ingress-canary-qxft6" Jul 09 18:54:38.721562 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721434 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88d1f57b-538f-4ef6-b0e8-616c9c493dab-ca-trust-extracted\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.721562 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721485 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2246db-5e70-45a8-94a5-4f387631fe03-config-volume\") pod \"dns-default-lvppt\" (UID: \"8f2246db-5e70-45a8-94a5-4f387631fe03\") " pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:38.721562 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721516 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjrg6\" (UniqueName: \"kubernetes.io/projected/8f2246db-5e70-45a8-94a5-4f387631fe03-kube-api-access-jjrg6\") pod \"dns-default-lvppt\" (UID: \"8f2246db-5e70-45a8-94a5-4f387631fe03\") " pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:38.721705 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721563 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f2246db-5e70-45a8-94a5-4f387631fe03-tmp-dir\") pod \"dns-default-lvppt\" (UID: \"8f2246db-5e70-45a8-94a5-4f387631fe03\") " pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:38.721705 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721659 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9293675-16ef-4401-af79-aa74f9597e8f-crio-socket\") pod \"insights-runtime-extractor-4wknv\" (UID: \"d9293675-16ef-4401-af79-aa74f9597e8f\") " pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.721705 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721689 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltppn\" (UniqueName: \"kubernetes.io/projected/d9293675-16ef-4401-af79-aa74f9597e8f-kube-api-access-ltppn\") pod \"insights-runtime-extractor-4wknv\" (UID: \"d9293675-16ef-4401-af79-aa74f9597e8f\") " pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.721853 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721754 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88d1f57b-538f-4ef6-b0e8-616c9c493dab-bound-sa-token\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.721853 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721812 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9293675-16ef-4401-af79-aa74f9597e8f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wknv\" (UID: \"d9293675-16ef-4401-af79-aa74f9597e8f\") " pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.721853 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721845 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88d1f57b-538f-4ef6-b0e8-616c9c493dab-installation-pull-secrets\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.722014 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721870 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88d1f57b-538f-4ef6-b0e8-616c9c493dab-ca-trust-extracted\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.722014 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721903 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88d1f57b-538f-4ef6-b0e8-616c9c493dab-registry-certificates\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.722014 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.721970 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88d1f57b-538f-4ef6-b0e8-616c9c493dab-trusted-ca\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.725235 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.725212 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88d1f57b-538f-4ef6-b0e8-616c9c493dab-registry-tls\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.725235 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.725225 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88d1f57b-538f-4ef6-b0e8-616c9c493dab-image-registry-private-configuration\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.725235 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.725214 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bd52903-4c1f-4840-a92b-c1ff27ec76be-cert\") pod \"ingress-canary-qxft6\" (UID: \"3bd52903-4c1f-4840-a92b-c1ff27ec76be\") " pod="openshift-ingress-canary/ingress-canary-qxft6" Jul 09 18:54:38.725443 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.725236 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88d1f57b-538f-4ef6-b0e8-616c9c493dab-installation-pull-secrets\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.730288 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.730256 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxgsn\" (UniqueName: \"kubernetes.io/projected/3bd52903-4c1f-4840-a92b-c1ff27ec76be-kube-api-access-pxgsn\") pod \"ingress-canary-qxft6\" (UID: \"3bd52903-4c1f-4840-a92b-c1ff27ec76be\") " pod="openshift-ingress-canary/ingress-canary-qxft6" Jul 09 18:54:38.730788 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.730757 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2tlm\" (UniqueName: \"kubernetes.io/projected/88d1f57b-538f-4ef6-b0e8-616c9c493dab-kube-api-access-b2tlm\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.731416 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.731395 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88d1f57b-538f-4ef6-b0e8-616c9c493dab-bound-sa-token\") pod \"image-registry-5dbb5989d9-7p7xp\" (UID: \"88d1f57b-538f-4ef6-b0e8-616c9c493dab\") " pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.822659 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.822617 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9293675-16ef-4401-af79-aa74f9597e8f-data-volume\") pod \"insights-runtime-extractor-4wknv\" (UID: \"d9293675-16ef-4401-af79-aa74f9597e8f\") " pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.822659 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.822665 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2246db-5e70-45a8-94a5-4f387631fe03-config-volume\") pod \"dns-default-lvppt\" (UID: \"8f2246db-5e70-45a8-94a5-4f387631fe03\") " pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:38.822859 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.822681 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjrg6\" (UniqueName: \"kubernetes.io/projected/8f2246db-5e70-45a8-94a5-4f387631fe03-kube-api-access-jjrg6\") pod \"dns-default-lvppt\" (UID: \"8f2246db-5e70-45a8-94a5-4f387631fe03\") " pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:38.822859 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.822698 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f2246db-5e70-45a8-94a5-4f387631fe03-tmp-dir\") pod \"dns-default-lvppt\" (UID: \"8f2246db-5e70-45a8-94a5-4f387631fe03\") " pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:38.822859 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.822716 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9293675-16ef-4401-af79-aa74f9597e8f-crio-socket\") pod \"insights-runtime-extractor-4wknv\" (UID: \"d9293675-16ef-4401-af79-aa74f9597e8f\") " pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.822859 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.822731 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltppn\" (UniqueName: \"kubernetes.io/projected/d9293675-16ef-4401-af79-aa74f9597e8f-kube-api-access-ltppn\") pod \"insights-runtime-extractor-4wknv\" (UID: \"d9293675-16ef-4401-af79-aa74f9597e8f\") " pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.823116 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.822900 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9293675-16ef-4401-af79-aa74f9597e8f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wknv\" (UID: \"d9293675-16ef-4401-af79-aa74f9597e8f\") " pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.823116 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.822935 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9293675-16ef-4401-af79-aa74f9597e8f-crio-socket\") pod \"insights-runtime-extractor-4wknv\" (UID: \"d9293675-16ef-4401-af79-aa74f9597e8f\") " pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.823116 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.822981 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f2246db-5e70-45a8-94a5-4f387631fe03-metrics-tls\") pod \"dns-default-lvppt\" (UID: \"8f2246db-5e70-45a8-94a5-4f387631fe03\") " pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:38.823116 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.823014 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9293675-16ef-4401-af79-aa74f9597e8f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wknv\" (UID: \"d9293675-16ef-4401-af79-aa74f9597e8f\") " pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.823116 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.823064 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9293675-16ef-4401-af79-aa74f9597e8f-data-volume\") pod \"insights-runtime-extractor-4wknv\" (UID: \"d9293675-16ef-4401-af79-aa74f9597e8f\") " pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.823116 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.823096 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f2246db-5e70-45a8-94a5-4f387631fe03-tmp-dir\") pod \"dns-default-lvppt\" (UID: \"8f2246db-5e70-45a8-94a5-4f387631fe03\") " pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:38.823416 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.823391 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2246db-5e70-45a8-94a5-4f387631fe03-config-volume\") pod \"dns-default-lvppt\" (UID: \"8f2246db-5e70-45a8-94a5-4f387631fe03\") " pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:38.823513 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.823490 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9293675-16ef-4401-af79-aa74f9597e8f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wknv\" (UID: \"d9293675-16ef-4401-af79-aa74f9597e8f\") " pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.825198 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.825169 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f2246db-5e70-45a8-94a5-4f387631fe03-metrics-tls\") pod \"dns-default-lvppt\" (UID: \"8f2246db-5e70-45a8-94a5-4f387631fe03\") " pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:38.825306 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.825249 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9293675-16ef-4401-af79-aa74f9597e8f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wknv\" (UID: \"d9293675-16ef-4401-af79-aa74f9597e8f\") " pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.831343 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.831317 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjrg6\" (UniqueName: \"kubernetes.io/projected/8f2246db-5e70-45a8-94a5-4f387631fe03-kube-api-access-jjrg6\") pod \"dns-default-lvppt\" (UID: \"8f2246db-5e70-45a8-94a5-4f387631fe03\") " pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:38.831571 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.831551 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltppn\" (UniqueName: \"kubernetes.io/projected/d9293675-16ef-4401-af79-aa74f9597e8f-kube-api-access-ltppn\") pod \"insights-runtime-extractor-4wknv\" (UID: \"d9293675-16ef-4401-af79-aa74f9597e8f\") " pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.859572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.859548 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsr6w" Jul 09 18:54:38.859682 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.859558 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:38.870746 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.870727 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qxft6" Jul 09 18:54:38.883695 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.883665 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4wknv" Jul 09 18:54:38.969406 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:38.969176 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:39.021935 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:39.021907 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qxft6"] Jul 09 18:54:39.025440 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:54:39.025400 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd52903_4c1f_4840_a92b_c1ff27ec76be.slice/crio-68d9fa49b882dd06152e02b47d9124fe4f68b5c4fe8e9a75b5d4f7d111a7d630 WatchSource:0}: Error finding container 68d9fa49b882dd06152e02b47d9124fe4f68b5c4fe8e9a75b5d4f7d111a7d630: Status 404 returned error can't find the container with id 68d9fa49b882dd06152e02b47d9124fe4f68b5c4fe8e9a75b5d4f7d111a7d630 Jul 09 18:54:39.031259 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:39.031044 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dbb5989d9-7p7xp"] Jul 09 18:54:39.034430 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:39.034347 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4wknv"] Jul 09 18:54:39.035047 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:54:39.035017 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d1f57b_538f_4ef6_b0e8_616c9c493dab.slice/crio-cb53958ee16a345a83fa589ce84096cdf5d0261be147411e47b4566256a1747f WatchSource:0}: Error finding container cb53958ee16a345a83fa589ce84096cdf5d0261be147411e47b4566256a1747f: Status 404 returned error can't find the container with id cb53958ee16a345a83fa589ce84096cdf5d0261be147411e47b4566256a1747f Jul 09 18:54:39.038576 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:54:39.038507 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9293675_16ef_4401_af79_aa74f9597e8f.slice/crio-176e82ae2fb56b8ece7d02d0efa9d81236d7a581381ac4f43b16e34c8cefbb7c WatchSource:0}: Error finding container 176e82ae2fb56b8ece7d02d0efa9d81236d7a581381ac4f43b16e34c8cefbb7c: Status 404 returned error can't find the container with id 176e82ae2fb56b8ece7d02d0efa9d81236d7a581381ac4f43b16e34c8cefbb7c Jul 09 18:54:39.114264 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:39.114031 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lvppt"] Jul 09 18:54:39.118375 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:54:39.118353 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2246db_5e70_45a8_94a5_4f387631fe03.slice/crio-4460585882faf6f22b9d63d90bc6309a02f914c83c2914386bd3ce9a9cfe73f4 WatchSource:0}: Error finding container 4460585882faf6f22b9d63d90bc6309a02f914c83c2914386bd3ce9a9cfe73f4: Status 404 returned error can't find the container with id 4460585882faf6f22b9d63d90bc6309a02f914c83c2914386bd3ce9a9cfe73f4 Jul 09 18:54:39.928912 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:39.928818 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wknv" event={"ID":"d9293675-16ef-4401-af79-aa74f9597e8f","Type":"ContainerStarted","Data":"176e82ae2fb56b8ece7d02d0efa9d81236d7a581381ac4f43b16e34c8cefbb7c"} Jul 09 18:54:39.933717 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:39.932860 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" event={"ID":"88d1f57b-538f-4ef6-b0e8-616c9c493dab","Type":"ContainerStarted","Data":"3a18a9c6f38a1af204bf21038c3926f97de140247d525e166d2116296647020c"} Jul 09 18:54:39.933717 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:39.932914 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" event={"ID":"88d1f57b-538f-4ef6-b0e8-616c9c493dab","Type":"ContainerStarted","Data":"cb53958ee16a345a83fa589ce84096cdf5d0261be147411e47b4566256a1747f"} Jul 09 18:54:39.933717 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:39.933651 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:54:39.935318 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:39.935292 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lvppt" event={"ID":"8f2246db-5e70-45a8-94a5-4f387631fe03","Type":"ContainerStarted","Data":"4460585882faf6f22b9d63d90bc6309a02f914c83c2914386bd3ce9a9cfe73f4"} Jul 09 18:54:39.937238 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:39.937214 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qxft6" event={"ID":"3bd52903-4c1f-4840-a92b-c1ff27ec76be","Type":"ContainerStarted","Data":"68d9fa49b882dd06152e02b47d9124fe4f68b5c4fe8e9a75b5d4f7d111a7d630"} Jul 09 18:54:39.957147 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:39.956885 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" podStartSLOduration=6.956865736 podStartE2EDuration="6.956865736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:39.955233482 +0000 UTC m=+64.780396520" watchObservedRunningTime="2026-07-09 18:54:39.956865736 +0000 UTC m=+64.782028775" Jul 09 18:54:41.344266 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:41.344228 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs\") pod \"network-metrics-daemon-kfvdz\" (UID: \"c46a7770-1771-4f8c-b35b-33ed971625c4\") " pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:41.348247 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:41.348022 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c46a7770-1771-4f8c-b35b-33ed971625c4-metrics-certs\") pod \"network-metrics-daemon-kfvdz\" (UID: \"c46a7770-1771-4f8c-b35b-33ed971625c4\") " pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:41.404079 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:41.404039 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfvdz" Jul 09 18:54:41.444960 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:41.444904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsgt\" (UniqueName: \"kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt\") pod \"network-check-target-4hsrp\" (UID: \"3f612d36-6ce3-44a2-9a90-61dd316a6703\") " pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:41.445106 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:41.445002 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:41.447554 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:41.447531 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8de73e1e-1dc7-4c83-98fd-ac2697ca4c15-original-pull-secret\") pod \"global-pull-secret-syncer-kss86\" (UID: \"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15\") " pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:41.447753 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:41.447733 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vsgt\" (UniqueName: \"kubernetes.io/projected/3f612d36-6ce3-44a2-9a90-61dd316a6703-kube-api-access-2vsgt\") pod \"network-check-target-4hsrp\" (UID: \"3f612d36-6ce3-44a2-9a90-61dd316a6703\") " pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:41.697842 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:41.697771 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kss86" Jul 09 18:54:41.708872 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:41.708848 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:45.004142 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:45.003842 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kss86"] Jul 09 18:54:45.023224 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:45.023022 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4hsrp"] Jul 09 18:54:45.028228 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:54:45.028198 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f612d36_6ce3_44a2_9a90_61dd316a6703.slice/crio-42cd6525657ecd616bd078e3bc5e93e387428882adaff2fe2b620c27f7db7c56 WatchSource:0}: Error finding container 42cd6525657ecd616bd078e3bc5e93e387428882adaff2fe2b620c27f7db7c56: Status 404 returned error can't find the container with id 42cd6525657ecd616bd078e3bc5e93e387428882adaff2fe2b620c27f7db7c56 Jul 09 18:54:45.033876 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:45.033853 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kfvdz"] Jul 09 18:54:45.039775 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:54:45.039738 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46a7770_1771_4f8c_b35b_33ed971625c4.slice/crio-83377683e1e2aa95bce776bd8b3ae5669cdc1d961257497c08d879cbd4cb2d3b WatchSource:0}: Error finding container 83377683e1e2aa95bce776bd8b3ae5669cdc1d961257497c08d879cbd4cb2d3b: Status 404 returned error can't find the container with id 83377683e1e2aa95bce776bd8b3ae5669cdc1d961257497c08d879cbd4cb2d3b Jul 09 18:54:45.967616 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:45.967377 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wknv" event={"ID":"d9293675-16ef-4401-af79-aa74f9597e8f","Type":"ContainerStarted","Data":"cc38ffdd84dbec7dcc4e556676dab97413f76f7eda5211bf69108f30c23c5db8"} Jul 09 18:54:45.967768 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:45.967661 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wknv" event={"ID":"d9293675-16ef-4401-af79-aa74f9597e8f","Type":"ContainerStarted","Data":"36ad65cddee717eeaf9bde46892fca8fdd3f23c1e1b29df1746647f88208d50c"} Jul 09 18:54:45.969043 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:45.968985 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kss86" event={"ID":"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15","Type":"ContainerStarted","Data":"677fbac8583e809a233cd12bf164e1026902773685111d87db356aa4c61c2fce"} Jul 09 18:54:45.970699 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:45.970671 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kfvdz" event={"ID":"c46a7770-1771-4f8c-b35b-33ed971625c4","Type":"ContainerStarted","Data":"83377683e1e2aa95bce776bd8b3ae5669cdc1d961257497c08d879cbd4cb2d3b"} Jul 09 18:54:45.972449 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:45.972421 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4hsrp" event={"ID":"3f612d36-6ce3-44a2-9a90-61dd316a6703","Type":"ContainerStarted","Data":"42cd6525657ecd616bd078e3bc5e93e387428882adaff2fe2b620c27f7db7c56"} Jul 09 18:54:45.975105 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:45.975081 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lvppt" event={"ID":"8f2246db-5e70-45a8-94a5-4f387631fe03","Type":"ContainerStarted","Data":"84b4dd9055dc8df24d12256b4adb495fd0f18d760e421c11fe167e5b9230137a"} Jul 09 18:54:45.975195 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:45.975127 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lvppt" event={"ID":"8f2246db-5e70-45a8-94a5-4f387631fe03","Type":"ContainerStarted","Data":"210d9bb31a5ad0687910aee69a279035af9fdd18de431e4506b4287916708f32"} Jul 09 18:54:45.975241 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:45.975209 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:45.976884 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:45.976843 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qxft6" event={"ID":"3bd52903-4c1f-4840-a92b-c1ff27ec76be","Type":"ContainerStarted","Data":"23e07750f10290485afc4c28841ecd85d968ea34975496a90e99dcd2a2ca2338"} Jul 09 18:54:45.997096 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:45.996590 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lvppt" podStartSLOduration=2.287824161 podStartE2EDuration="7.996570166s" totalImagesPullingTime="5.708746005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:45.995961656 +0000 UTC m=+70.821124685" watchObservedRunningTime="2026-07-09 18:54:45.996570166 +0000 UTC m=+70.821733203" Jul 09 18:54:46.971248 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.969938 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qxft6" podStartSLOduration=3.174585799 podStartE2EDuration="8.969918663s" totalImagesPullingTime="5.795332864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:46.013998973 +0000 UTC m=+70.839162014" watchObservedRunningTime="2026-07-09 18:54:46.969918663 +0000 UTC m=+71.795081724" Jul 09 18:54:46.971248 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.970170 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-h27h2"] Jul 09 18:54:46.972464 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.972444 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:46.976243 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.976223 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 09 18:54:46.976898 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.976881 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 18:54:46.978094 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.978073 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 18:54:46.978567 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.978550 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 09 18:54:46.978646 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.978600 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nskg4\"" Jul 09 18:54:46.978687 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.978556 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 18:54:46.978754 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.978743 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 09 18:54:46.982935 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.982868 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kfvdz" event={"ID":"c46a7770-1771-4f8c-b35b-33ed971625c4","Type":"ContainerStarted","Data":"8d52e58f655b19eba8c9deee4efd73bdea78129ec00bd94aae0619f711c012d9"} Jul 09 18:54:46.982935 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.982921 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kfvdz" event={"ID":"c46a7770-1771-4f8c-b35b-33ed971625c4","Type":"ContainerStarted","Data":"ae6df279302249eb247a5c9789f33baef1200968466c79435e82cf5797d51983"} Jul 09 18:54:46.993745 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.993723 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-h27h2"] Jul 09 18:54:46.995156 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.995135 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-54jsv"] Jul 09 18:54:46.997384 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.997364 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:46.999915 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:46.999896 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7vtbj\"" Jul 09 18:54:47.000034 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.000016 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 18:54:47.000095 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.000061 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 18:54:47.000153 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.000113 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 18:54:47.046341 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.046283 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kfvdz" podStartSLOduration=71.108368377 podStartE2EDuration="1m12.046263319s" totalImagesPullingTime="937.894942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:47.046239079 +0000 UTC m=+71.871402117" watchObservedRunningTime="2026-07-09 18:54:47.046263319 +0000 UTC m=+71.871426354" Jul 09 18:54:47.086134 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.086100 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72fec418-967a-4a13-84d1-2a0a141251be-node-exporter-tls\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.086313 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.086143 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74defd5b-37e2-48c7-9b3c-849720978d12-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.086313 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.086202 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72fec418-967a-4a13-84d1-2a0a141251be-sys\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.086313 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.086227 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/72fec418-967a-4a13-84d1-2a0a141251be-node-exporter-wtmp\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.086313 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.086254 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74defd5b-37e2-48c7-9b3c-849720978d12-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.086313 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.086292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/74defd5b-37e2-48c7-9b3c-849720978d12-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.086589 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.086336 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/72fec418-967a-4a13-84d1-2a0a141251be-node-exporter-textfile\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.086589 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.086355 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74defd5b-37e2-48c7-9b3c-849720978d12-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.086589 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.086381 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/74defd5b-37e2-48c7-9b3c-849720978d12-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.086589 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.086446 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8hw7\" (UniqueName: \"kubernetes.io/projected/74defd5b-37e2-48c7-9b3c-849720978d12-kube-api-access-g8hw7\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.086589 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.086487 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72fec418-967a-4a13-84d1-2a0a141251be-metrics-client-ca\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.086589 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.086523 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72fec418-967a-4a13-84d1-2a0a141251be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.086589 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.086558 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/72fec418-967a-4a13-84d1-2a0a141251be-node-exporter-accelerators-collector-config\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.086589 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.086588 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/72fec418-967a-4a13-84d1-2a0a141251be-root\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.087011 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.086622 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8jzv\" (UniqueName: \"kubernetes.io/projected/72fec418-967a-4a13-84d1-2a0a141251be-kube-api-access-j8jzv\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.187293 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187255 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72fec418-967a-4a13-84d1-2a0a141251be-sys\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.187293 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187304 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/72fec418-967a-4a13-84d1-2a0a141251be-node-exporter-wtmp\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.187534 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187331 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74defd5b-37e2-48c7-9b3c-849720978d12-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.187534 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187363 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/74defd5b-37e2-48c7-9b3c-849720978d12-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.187534 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187402 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/72fec418-967a-4a13-84d1-2a0a141251be-node-exporter-textfile\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.187534 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187431 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74defd5b-37e2-48c7-9b3c-849720978d12-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.187534 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187459 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/74defd5b-37e2-48c7-9b3c-849720978d12-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.187534 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187496 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8hw7\" (UniqueName: \"kubernetes.io/projected/74defd5b-37e2-48c7-9b3c-849720978d12-kube-api-access-g8hw7\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.187534 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187518 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72fec418-967a-4a13-84d1-2a0a141251be-metrics-client-ca\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.187853 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187751 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/72fec418-967a-4a13-84d1-2a0a141251be-node-exporter-wtmp\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.187853 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187810 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72fec418-967a-4a13-84d1-2a0a141251be-sys\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.188555 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187868 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72fec418-967a-4a13-84d1-2a0a141251be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.188555 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187882 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/72fec418-967a-4a13-84d1-2a0a141251be-node-exporter-textfile\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.188555 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:47.187896 2569 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 09 18:54:47.188555 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187902 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/72fec418-967a-4a13-84d1-2a0a141251be-node-exporter-accelerators-collector-config\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.188555 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187931 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/72fec418-967a-4a13-84d1-2a0a141251be-root\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.188555 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.187938 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/74defd5b-37e2-48c7-9b3c-849720978d12-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.188555 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:47.187990 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74defd5b-37e2-48c7-9b3c-849720978d12-kube-state-metrics-tls podName:74defd5b-37e2-48c7-9b3c-849720978d12 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:47.687971689 +0000 UTC m=+72.513134704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/74defd5b-37e2-48c7-9b3c-849720978d12-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-h27h2" (UID: "74defd5b-37e2-48c7-9b3c-849720978d12") : secret "kube-state-metrics-tls" not found Jul 09 18:54:47.188555 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.188011 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8jzv\" (UniqueName: \"kubernetes.io/projected/72fec418-967a-4a13-84d1-2a0a141251be-kube-api-access-j8jzv\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.188555 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.188022 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/72fec418-967a-4a13-84d1-2a0a141251be-root\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.188555 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.188039 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72fec418-967a-4a13-84d1-2a0a141251be-node-exporter-tls\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.188555 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.188069 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74defd5b-37e2-48c7-9b3c-849720978d12-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.188555 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.188443 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/74defd5b-37e2-48c7-9b3c-849720978d12-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.188555 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.188479 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72fec418-967a-4a13-84d1-2a0a141251be-metrics-client-ca\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.189192 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.188536 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/72fec418-967a-4a13-84d1-2a0a141251be-node-exporter-accelerators-collector-config\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.189192 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.188841 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74defd5b-37e2-48c7-9b3c-849720978d12-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.192993 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.192911 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72fec418-967a-4a13-84d1-2a0a141251be-node-exporter-tls\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.193512 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.193487 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74defd5b-37e2-48c7-9b3c-849720978d12-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.194088 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.194048 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72fec418-967a-4a13-84d1-2a0a141251be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.199798 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.199770 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8jzv\" (UniqueName: \"kubernetes.io/projected/72fec418-967a-4a13-84d1-2a0a141251be-kube-api-access-j8jzv\") pod \"node-exporter-54jsv\" (UID: \"72fec418-967a-4a13-84d1-2a0a141251be\") " pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.200049 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.200029 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8hw7\" (UniqueName: \"kubernetes.io/projected/74defd5b-37e2-48c7-9b3c-849720978d12-kube-api-access-g8hw7\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.307268 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.307198 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-54jsv" Jul 09 18:54:47.691676 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:47.691584 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74defd5b-37e2-48c7-9b3c-849720978d12-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:47.691837 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:47.691736 2569 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 09 18:54:47.691837 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:47.691795 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74defd5b-37e2-48c7-9b3c-849720978d12-kube-state-metrics-tls podName:74defd5b-37e2-48c7-9b3c-849720978d12 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:48.69177633 +0000 UTC m=+73.516939350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/74defd5b-37e2-48c7-9b3c-849720978d12-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-h27h2" (UID: "74defd5b-37e2-48c7-9b3c-849720978d12") : secret "kube-state-metrics-tls" not found Jul 09 18:54:48.196718 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.196682 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 18:54:48.204610 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.204584 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.208614 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.208311 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 09 18:54:48.208614 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.208354 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 09 18:54:48.208614 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.208408 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 09 18:54:48.208614 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.208311 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 09 18:54:48.208614 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.208312 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 09 18:54:48.208614 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.208608 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 09 18:54:48.209036 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.208681 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 09 18:54:48.209036 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.208718 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5ddg2\"" Jul 09 18:54:48.209036 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.208763 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 09 18:54:48.209036 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.208966 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 09 18:54:48.219766 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.219744 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 18:54:48.296902 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.296830 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cd93c96-1509-4341-bac4-67f67b8ad24d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.296902 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.296901 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.297164 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.296940 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7cd93c96-1509-4341-bac4-67f67b8ad24d-config-out\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.297164 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.296984 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7cd93c96-1509-4341-bac4-67f67b8ad24d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.297164 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.297053 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-web-config\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.297164 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.297095 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cd93c96-1509-4341-bac4-67f67b8ad24d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.297164 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.297134 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.297398 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.297165 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxb44\" (UniqueName: \"kubernetes.io/projected/7cd93c96-1509-4341-bac4-67f67b8ad24d-kube-api-access-sxb44\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.297398 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.297197 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.297398 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.297276 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.297398 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.297318 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7cd93c96-1509-4341-bac4-67f67b8ad24d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.297398 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.297347 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-config-volume\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.297578 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.297408 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.398135 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.398099 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxb44\" (UniqueName: \"kubernetes.io/projected/7cd93c96-1509-4341-bac4-67f67b8ad24d-kube-api-access-sxb44\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.398314 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.398142 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.398314 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.398182 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.398314 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.398205 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7cd93c96-1509-4341-bac4-67f67b8ad24d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.398314 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.398221 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-config-volume\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.398314 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.398251 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.398314 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.398283 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cd93c96-1509-4341-bac4-67f67b8ad24d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.398743 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.398709 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7cd93c96-1509-4341-bac4-67f67b8ad24d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.398822 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.398731 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.398822 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.398798 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7cd93c96-1509-4341-bac4-67f67b8ad24d-config-out\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.398923 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:48.398827 2569 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 09 18:54:48.398923 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:54:48.398892 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-main-tls podName:7cd93c96-1509-4341-bac4-67f67b8ad24d nodeName:}" failed. No retries permitted until 2026-07-09 18:54:48.898871167 +0000 UTC m=+73.724034188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "7cd93c96-1509-4341-bac4-67f67b8ad24d") : secret "alertmanager-main-tls" not found Jul 09 18:54:48.399592 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.398832 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7cd93c96-1509-4341-bac4-67f67b8ad24d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.399592 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.399168 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-web-config\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.399592 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.399198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cd93c96-1509-4341-bac4-67f67b8ad24d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.399592 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.399238 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.399592 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.399506 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cd93c96-1509-4341-bac4-67f67b8ad24d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.401396 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.401369 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cd93c96-1509-4341-bac4-67f67b8ad24d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.401681 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.401648 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7cd93c96-1509-4341-bac4-67f67b8ad24d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.402335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.402293 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-web-config\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.402865 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.402831 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.402982 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.402926 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.403167 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.403144 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7cd93c96-1509-4341-bac4-67f67b8ad24d-config-out\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.403555 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.403484 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-config-volume\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.403719 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.403701 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.404401 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.404383 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.409366 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.409343 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxb44\" (UniqueName: \"kubernetes.io/projected/7cd93c96-1509-4341-bac4-67f67b8ad24d-kube-api-access-sxb44\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.702152 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.701934 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74defd5b-37e2-48c7-9b3c-849720978d12-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:48.704609 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.704583 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74defd5b-37e2-48c7-9b3c-849720978d12-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-h27h2\" (UID: \"74defd5b-37e2-48c7-9b3c-849720978d12\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:48.784190 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.784154 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" Jul 09 18:54:48.903988 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.903927 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:48.906896 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:48.906874 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:49.119097 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:49.119058 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:54:50.839367 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:54:50.839335 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72fec418_967a_4a13_84d1_2a0a141251be.slice/crio-380a219c6c27fbdeadbc77f15475e511d90d4665539b5060bd2c6625df1cb418 WatchSource:0}: Error finding container 380a219c6c27fbdeadbc77f15475e511d90d4665539b5060bd2c6625df1cb418: Status 404 returned error can't find the container with id 380a219c6c27fbdeadbc77f15475e511d90d4665539b5060bd2c6625df1cb418 Jul 09 18:54:50.998983 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:50.998804 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kss86" event={"ID":"8de73e1e-1dc7-4c83-98fd-ac2697ca4c15","Type":"ContainerStarted","Data":"1291b17610c08d29cdb67089e8e2a6b523192c9f67dcd802bc6faf634379d727"} Jul 09 18:54:51.001212 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:51.000975 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4hsrp" event={"ID":"3f612d36-6ce3-44a2-9a90-61dd316a6703","Type":"ContainerStarted","Data":"9355cd23e025e7e1b32214b36830c014bc6626299a5636b569d669333cdbd0c0"} Jul 09 18:54:51.001212 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:51.001061 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:54:51.003749 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:51.003723 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wknv" event={"ID":"d9293675-16ef-4401-af79-aa74f9597e8f","Type":"ContainerStarted","Data":"addbdd4307ded593b5e1e643ef83905378fa2b2d4a3ebdfe3bfbb7ffaf03e2e1"} Jul 09 18:54:51.004488 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:51.004467 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-h27h2"] Jul 09 18:54:51.004803 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:51.004786 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-54jsv" event={"ID":"72fec418-967a-4a13-84d1-2a0a141251be","Type":"ContainerStarted","Data":"380a219c6c27fbdeadbc77f15475e511d90d4665539b5060bd2c6625df1cb418"} Jul 09 18:54:51.008306 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:54:51.008274 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74defd5b_37e2_48c7_9b3c_849720978d12.slice/crio-7b247a5ae8d7fda9fd3ec587b04fa5600ffa15de9b9653a240ec49f2b701d03a WatchSource:0}: Error finding container 7b247a5ae8d7fda9fd3ec587b04fa5600ffa15de9b9653a240ec49f2b701d03a: Status 404 returned error can't find the container with id 7b247a5ae8d7fda9fd3ec587b04fa5600ffa15de9b9653a240ec49f2b701d03a Jul 09 18:54:51.010684 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:51.010303 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 18:54:51.012444 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:54:51.012426 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd93c96_1509_4341_bac4_67f67b8ad24d.slice/crio-2d12ddda1dd329b038b1c538ad67b49a05da28186e90ea571af2801b3a15248f WatchSource:0}: Error finding container 2d12ddda1dd329b038b1c538ad67b49a05da28186e90ea571af2801b3a15248f: Status 404 returned error can't find the container with id 2d12ddda1dd329b038b1c538ad67b49a05da28186e90ea571af2801b3a15248f Jul 09 18:54:51.026145 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:51.026099 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kss86" podStartSLOduration=70.18944529 podStartE2EDuration="1m16.026080958s" totalImagesPullingTime="5.836635668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:51.025191776 +0000 UTC m=+75.850354812" watchObservedRunningTime="2026-07-09 18:54:51.026080958 +0000 UTC m=+75.851243994" Jul 09 18:54:51.047090 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:51.042704 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4hsrp" podStartSLOduration=70.22993372 podStartE2EDuration="1m16.042683911s" totalImagesPullingTime="5.812750191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:51.04127917 +0000 UTC m=+75.866442209" watchObservedRunningTime="2026-07-09 18:54:51.042683911 +0000 UTC m=+75.867846949" Jul 09 18:54:51.073219 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:51.073114 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4wknv" podStartSLOduration=1.448155758 podStartE2EDuration="13.073100535s" totalImagesPullingTime="11.624944777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:54:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:51.072038995 +0000 UTC m=+75.897202026" watchObservedRunningTime="2026-07-09 18:54:51.073100535 +0000 UTC m=+75.898263572" Jul 09 18:54:52.011404 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.011370 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cd93c96-1509-4341-bac4-67f67b8ad24d","Type":"ContainerStarted","Data":"2d12ddda1dd329b038b1c538ad67b49a05da28186e90ea571af2801b3a15248f"} Jul 09 18:54:52.012655 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.012629 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" event={"ID":"74defd5b-37e2-48c7-9b3c-849720978d12","Type":"ContainerStarted","Data":"7b247a5ae8d7fda9fd3ec587b04fa5600ffa15de9b9653a240ec49f2b701d03a"} Jul 09 18:54:52.014498 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.014470 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-54jsv" event={"ID":"72fec418-967a-4a13-84d1-2a0a141251be","Type":"ContainerStarted","Data":"563c41bfd37e52d3178fd771bf49836863e01371b480c85f29206cfde42b2944"} Jul 09 18:54:52.245811 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.245739 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-b5f89968d-j7zhv"] Jul 09 18:54:52.248408 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.248381 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.251302 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.251273 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 09 18:54:52.251302 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.251289 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 09 18:54:52.251613 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.251594 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 09 18:54:52.252154 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.251918 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 09 18:54:52.252154 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.251933 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-s2kq5\"" Jul 09 18:54:52.252330 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.252304 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 09 18:54:52.259060 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.259039 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 09 18:54:52.269840 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.269775 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b5f89968d-j7zhv"] Jul 09 18:54:52.330967 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.330928 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-metrics-client-ca\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.331069 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.331026 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.331144 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.331072 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-serving-certs-ca-bundle\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.331144 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.331102 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-secret-telemeter-client\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.331239 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.331141 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cshnb\" (UniqueName: \"kubernetes.io/projected/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-kube-api-access-cshnb\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.331281 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.331250 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-telemeter-client-tls\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.331318 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.331287 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.331422 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.331322 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-federate-client-tls\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.347637 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.347440 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d4b9c584f-g2s7b"] Jul 09 18:54:52.349625 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.349608 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.353927 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.353845 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 18:54:52.353927 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.353908 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 18:54:52.354092 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.353972 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 18:54:52.354738 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.354284 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 18:54:52.354738 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.354366 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 18:54:52.354738 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.354374 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 18:54:52.354738 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.354685 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 18:54:52.355014 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.354770 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cplfs\"" Jul 09 18:54:52.358880 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.358858 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 18:54:52.362318 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.362300 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4b9c584f-g2s7b"] Jul 09 18:54:52.432533 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.432507 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-serving-certs-ca-bundle\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.432653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.432538 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-secret-telemeter-client\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.432653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.432561 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshnb\" (UniqueName: \"kubernetes.io/projected/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-kube-api-access-cshnb\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.432653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.432586 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-oauth-serving-cert\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.432653 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.432629 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-config\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.432785 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.432668 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-telemeter-client-tls\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.432785 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.432702 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.432785 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.432745 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-federate-client-tls\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.432785 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.432777 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-serving-cert\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.432895 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.432800 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-oauth-config\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.432895 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.432822 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-trusted-ca-bundle\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.432895 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.432867 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdq8b\" (UniqueName: \"kubernetes.io/projected/f12e392b-c4df-4a56-8bb4-75593ef65bb5-kube-api-access-mdq8b\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.433040 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.432891 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-service-ca\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.433040 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.432941 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-metrics-client-ca\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.433040 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.432989 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.434462 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.434044 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-serving-certs-ca-bundle\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.434603 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.434570 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-metrics-client-ca\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.437541 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.436780 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.437541 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.437245 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.437541 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.437336 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-secret-telemeter-client\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.437993 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.437970 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-federate-client-tls\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.438125 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.438090 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-telemeter-client-tls\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.445654 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.445626 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshnb\" (UniqueName: \"kubernetes.io/projected/0abfb907-fdb2-40df-bc7f-fc5e3da9a86d-kube-api-access-cshnb\") pod \"telemeter-client-b5f89968d-j7zhv\" (UID: \"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d\") " pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.534027 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.533993 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-config\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.534196 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.534052 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-serving-cert\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.534196 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.534075 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-oauth-config\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.534196 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.534095 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-trusted-ca-bundle\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.534196 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.534120 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdq8b\" (UniqueName: \"kubernetes.io/projected/f12e392b-c4df-4a56-8bb4-75593ef65bb5-kube-api-access-mdq8b\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.534196 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.534145 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-service-ca\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.534449 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.534217 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-oauth-serving-cert\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.535104 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.534821 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-config\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.535104 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.534900 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-oauth-serving-cert\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.535104 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.535064 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-trusted-ca-bundle\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.535347 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.535157 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-service-ca\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.536773 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.536752 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-oauth-config\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.537028 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.537006 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-serving-cert\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.544354 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.544330 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdq8b\" (UniqueName: \"kubernetes.io/projected/f12e392b-c4df-4a56-8bb4-75593ef65bb5-kube-api-access-mdq8b\") pod \"console-7d4b9c584f-g2s7b\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.560909 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.560884 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" Jul 09 18:54:52.676556 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.676446 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:54:52.883623 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.883599 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4b9c584f-g2s7b"] Jul 09 18:54:52.886339 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:54:52.886317 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12e392b_c4df_4a56_8bb4_75593ef65bb5.slice/crio-307a804f3b62aefa126b12961b20b3f27d48fa09961a0e941329ede1b685487e WatchSource:0}: Error finding container 307a804f3b62aefa126b12961b20b3f27d48fa09961a0e941329ede1b685487e: Status 404 returned error can't find the container with id 307a804f3b62aefa126b12961b20b3f27d48fa09961a0e941329ede1b685487e Jul 09 18:54:52.897402 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:52.897341 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b5f89968d-j7zhv"] Jul 09 18:54:52.900813 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:54:52.900792 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abfb907_fdb2_40df_bc7f_fc5e3da9a86d.slice/crio-6d92db38c07578f2b47033b90d684394bcaf7e95900df4f19a2002845f09df77 WatchSource:0}: Error finding container 6d92db38c07578f2b47033b90d684394bcaf7e95900df4f19a2002845f09df77: Status 404 returned error can't find the container with id 6d92db38c07578f2b47033b90d684394bcaf7e95900df4f19a2002845f09df77 Jul 09 18:54:53.019485 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:53.019455 2569 generic.go:356] "Generic (PLEG): container finished" podID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerID="42bd535575dc57b2c1ff5ac62536d0862ba4c7a8d4e89cb6aecc81cfcd8cdb86" exitCode=0 Jul 09 18:54:53.019838 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:53.019538 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cd93c96-1509-4341-bac4-67f67b8ad24d","Type":"ContainerDied","Data":"42bd535575dc57b2c1ff5ac62536d0862ba4c7a8d4e89cb6aecc81cfcd8cdb86"} Jul 09 18:54:53.024262 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:53.024238 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" event={"ID":"74defd5b-37e2-48c7-9b3c-849720978d12","Type":"ContainerStarted","Data":"a589ec4ddb34697121dba4c372788c4f49c4daf46f7f51ff8ec030e0022b798b"} Jul 09 18:54:53.024374 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:53.024302 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" event={"ID":"74defd5b-37e2-48c7-9b3c-849720978d12","Type":"ContainerStarted","Data":"007adb6cc7069a0beaf164b8a723f5994c31cc2e616dffd6e341280775fc7b81"} Jul 09 18:54:53.025823 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:53.025800 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4b9c584f-g2s7b" event={"ID":"f12e392b-c4df-4a56-8bb4-75593ef65bb5","Type":"ContainerStarted","Data":"307a804f3b62aefa126b12961b20b3f27d48fa09961a0e941329ede1b685487e"} Jul 09 18:54:53.027588 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:53.027570 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" event={"ID":"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d","Type":"ContainerStarted","Data":"6d92db38c07578f2b47033b90d684394bcaf7e95900df4f19a2002845f09df77"} Jul 09 18:54:53.029919 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:53.029895 2569 generic.go:356] "Generic (PLEG): container finished" podID="72fec418-967a-4a13-84d1-2a0a141251be" containerID="563c41bfd37e52d3178fd771bf49836863e01371b480c85f29206cfde42b2944" exitCode=0 Jul 09 18:54:53.030039 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:53.029931 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-54jsv" event={"ID":"72fec418-967a-4a13-84d1-2a0a141251be","Type":"ContainerDied","Data":"563c41bfd37e52d3178fd771bf49836863e01371b480c85f29206cfde42b2944"} Jul 09 18:54:53.030039 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:53.029985 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-54jsv" event={"ID":"72fec418-967a-4a13-84d1-2a0a141251be","Type":"ContainerStarted","Data":"458fc37add3acaa9d50bd023a30168f2c020b60baa510670bc283350d8a84627"} Jul 09 18:54:53.030039 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:53.030002 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-54jsv" event={"ID":"72fec418-967a-4a13-84d1-2a0a141251be","Type":"ContainerStarted","Data":"fab31de8a785499c9d877dd8ce715da808e5e2e03cc804f97b4a5f1af7264640"} Jul 09 18:54:53.084305 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:53.084257 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-54jsv" podStartSLOduration=6.010775159 podStartE2EDuration="7.084239009s" totalImagesPullingTime="991.807531ms" totalInitContainerRuntime="81.656319ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:53.083070071 +0000 UTC m=+77.908233109" watchObservedRunningTime="2026-07-09 18:54:53.084239009 +0000 UTC m=+77.909402062" Jul 09 18:54:54.039426 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:54.039376 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" event={"ID":"74defd5b-37e2-48c7-9b3c-849720978d12","Type":"ContainerStarted","Data":"1937f8b44b372c0c822d8be1527fce4dd943ef8d175545118de6fec7bbcbb98b"} Jul 09 18:54:54.064519 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:54.064329 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-h27h2" podStartSLOduration=6.286547617 podStartE2EDuration="8.064308573s" totalImagesPullingTime="1.777760956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:54.063254229 +0000 UTC m=+78.888417258" watchObservedRunningTime="2026-07-09 18:54:54.064308573 +0000 UTC m=+78.889471611" Jul 09 18:54:55.050166 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:55.050033 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cd93c96-1509-4341-bac4-67f67b8ad24d","Type":"ContainerStarted","Data":"ef5d0bec8c132ee5526ad9138e8b901d117fc51f916ca7ba83f82312cd1dd369"} Jul 09 18:54:55.050166 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:55.050135 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cd93c96-1509-4341-bac4-67f67b8ad24d","Type":"ContainerStarted","Data":"e6aebd81828216a8f2a792a40c45a56a31767244c276494aa31221abbcef53d9"} Jul 09 18:54:55.050166 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:55.050171 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cd93c96-1509-4341-bac4-67f67b8ad24d","Type":"ContainerStarted","Data":"c256d913686501087f45ce7eb0578e02735dc72403b3b7ccbcab5d603f6ea55b"} Jul 09 18:54:55.985476 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:55.985438 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lvppt" Jul 09 18:54:57.061110 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:57.061046 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cd93c96-1509-4341-bac4-67f67b8ad24d","Type":"ContainerStarted","Data":"c6742370d24c11d07efba7470dff902f20c274c19ef1e818dce75f6da7a30876"} Jul 09 18:54:57.061528 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:57.061155 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cd93c96-1509-4341-bac4-67f67b8ad24d","Type":"ContainerStarted","Data":"5703e42bcd8aeeea740218130c14835f6a9c15c1572844080e22657c4abbd74a"} Jul 09 18:54:57.062622 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:57.062571 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4b9c584f-g2s7b" event={"ID":"f12e392b-c4df-4a56-8bb4-75593ef65bb5","Type":"ContainerStarted","Data":"b4e0676f371f69909918297e674e82c9c25b50128b0246caf24a6e14cc8ce013"} Jul 09 18:54:57.064682 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:57.064656 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" event={"ID":"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d","Type":"ContainerStarted","Data":"5f1ab3c5b39cd72514d3eb7aa5e51f88788e2f0cc03e47b41940ef523e9aa00d"} Jul 09 18:54:57.064780 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:57.064699 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" event={"ID":"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d","Type":"ContainerStarted","Data":"c36c34ac39d091a830e695a76a5324fe53902590d2d0da4e8e0f13cc4154e6db"} Jul 09 18:54:57.064780 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:57.064719 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" event={"ID":"0abfb907-fdb2-40df-bc7f-fc5e3da9a86d","Type":"ContainerStarted","Data":"50665cb8e1cf6ee2b1b6018a898bc5fd75d8d0febb1d5a4fe46773d3edac14ae"} Jul 09 18:54:57.085046 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:57.084999 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d4b9c584f-g2s7b" podStartSLOduration=1.351441328 podStartE2EDuration="5.084985844s" totalImagesPullingTime="3.733544516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:57.083807365 +0000 UTC m=+81.908970619" watchObservedRunningTime="2026-07-09 18:54:57.084985844 +0000 UTC m=+81.910148880" Jul 09 18:54:57.107848 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:57.107801 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-b5f89968d-j7zhv" podStartSLOduration=1.384357959 podStartE2EDuration="5.10778131s" totalImagesPullingTime="3.723423351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:57.105637965 +0000 UTC m=+81.930801002" watchObservedRunningTime="2026-07-09 18:54:57.10778131 +0000 UTC m=+81.932944351" Jul 09 18:54:58.044268 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.044238 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d4b9c584f-g2s7b"] Jul 09 18:54:58.071289 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.071249 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cd93c96-1509-4341-bac4-67f67b8ad24d","Type":"ContainerStarted","Data":"f719066d12070dd96ddd128d5db091af271a36f36ae57b80fcc6cc71edfe4443"} Jul 09 18:54:58.083262 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.083232 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75b5cc9556-x9gwz"] Jul 09 18:54:58.086511 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.086493 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.097655 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.097631 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75b5cc9556-x9gwz"] Jul 09 18:54:58.104367 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.104327 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.780737999 podStartE2EDuration="10.104312555s" totalImagesPullingTime="4.300700457s" totalInitContainerRuntime="22.874099ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:58.1027467 +0000 UTC m=+82.927909739" watchObservedRunningTime="2026-07-09 18:54:58.104312555 +0000 UTC m=+82.929475592" Jul 09 18:54:58.184838 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.184804 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-config\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.185029 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.184972 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-trusted-ca-bundle\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.185100 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.185038 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-oauth-serving-cert\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.185100 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.185069 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5726\" (UniqueName: \"kubernetes.io/projected/ca815a25-029a-4db9-8254-1b1e891bfeeb-kube-api-access-q5726\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.185225 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.185175 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-serving-cert\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.185343 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.185244 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-oauth-config\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.185543 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.185515 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-service-ca\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.287162 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.287112 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-config\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.287162 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.287169 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-trusted-ca-bundle\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.287376 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.287195 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-oauth-serving-cert\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.287376 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.287222 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5726\" (UniqueName: \"kubernetes.io/projected/ca815a25-029a-4db9-8254-1b1e891bfeeb-kube-api-access-q5726\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.287376 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.287271 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-serving-cert\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.287376 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.287321 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-oauth-config\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.287556 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.287389 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-service-ca\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.287928 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.287898 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-config\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.288097 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.288054 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-oauth-serving-cert\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.288194 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.288172 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-service-ca\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.288372 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.288355 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-trusted-ca-bundle\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.289757 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.289734 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-serving-cert\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.289847 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.289770 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-oauth-config\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.296272 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.296252 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5726\" (UniqueName: \"kubernetes.io/projected/ca815a25-029a-4db9-8254-1b1e891bfeeb-kube-api-access-q5726\") pod \"console-75b5cc9556-x9gwz\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.394745 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.394715 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:54:58.535112 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:58.535080 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75b5cc9556-x9gwz"] Jul 09 18:54:58.537637 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:54:58.537584 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca815a25_029a_4db9_8254_1b1e891bfeeb.slice/crio-1bebd540f7401861dcd9e176f3cdb18e983ee42ae73d3e2be479d6ed376bea28 WatchSource:0}: Error finding container 1bebd540f7401861dcd9e176f3cdb18e983ee42ae73d3e2be479d6ed376bea28: Status 404 returned error can't find the container with id 1bebd540f7401861dcd9e176f3cdb18e983ee42ae73d3e2be479d6ed376bea28 Jul 09 18:54:59.076378 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:59.076343 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75b5cc9556-x9gwz" event={"ID":"ca815a25-029a-4db9-8254-1b1e891bfeeb","Type":"ContainerStarted","Data":"dfe5ce2b97706be2beb7221a504c1c05d806d0d0ff4db1f3f449c287936a3595"} Jul 09 18:54:59.076811 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:59.076389 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75b5cc9556-x9gwz" event={"ID":"ca815a25-029a-4db9-8254-1b1e891bfeeb","Type":"ContainerStarted","Data":"1bebd540f7401861dcd9e176f3cdb18e983ee42ae73d3e2be479d6ed376bea28"} Jul 09 18:54:59.096846 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:54:59.096797 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75b5cc9556-x9gwz" podStartSLOduration=1.09677918 podStartE2EDuration="1.09677918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:59.09544879 +0000 UTC m=+83.920611830" watchObservedRunningTime="2026-07-09 18:54:59.09677918 +0000 UTC m=+83.921942196" Jul 09 18:55:01.948290 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:01.948262 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5dbb5989d9-7p7xp" Jul 09 18:55:02.676605 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:02.676571 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:55:08.394900 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:08.394859 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:55:08.395341 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:08.394942 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:55:08.399833 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:08.399811 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:55:09.116464 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:09.116438 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:55:15.077920 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:15.077890 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qxft6_3bd52903-4c1f-4840-a92b-c1ff27ec76be/serve-healthcheck-canary/0.log" Jul 09 18:55:22.017706 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:22.017675 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4hsrp" Jul 09 18:55:24.095832 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:24.095786 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d4b9c584f-g2s7b" podUID="f12e392b-c4df-4a56-8bb4-75593ef65bb5" containerName="console" containerID="cri-o://b4e0676f371f69909918297e674e82c9c25b50128b0246caf24a6e14cc8ce013" gracePeriod=15 Jul 09 18:55:24.871932 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:24.871903 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:55:24.999901 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:24.999821 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-config\") pod \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " Jul 09 18:55:24.999901 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:24.999854 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-serving-cert\") pod \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " Jul 09 18:55:24.999901 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:24.999879 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-oauth-serving-cert\") pod \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " Jul 09 18:55:25.000190 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:24.999914 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-service-ca\") pod \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " Jul 09 18:55:25.000190 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.000067 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-trusted-ca-bundle\") pod \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " Jul 09 18:55:25.000190 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.000134 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdq8b\" (UniqueName: \"kubernetes.io/projected/f12e392b-c4df-4a56-8bb4-75593ef65bb5-kube-api-access-mdq8b\") pod \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " Jul 09 18:55:25.000335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.000191 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-oauth-config\") pod \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\" (UID: \"f12e392b-c4df-4a56-8bb4-75593ef65bb5\") " Jul 09 18:55:25.000380 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.000330 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f12e392b-c4df-4a56-8bb4-75593ef65bb5" (UID: "f12e392b-c4df-4a56-8bb4-75593ef65bb5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:55:25.000380 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.000355 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-config" (OuterVolumeSpecName: "console-config") pod "f12e392b-c4df-4a56-8bb4-75593ef65bb5" (UID: "f12e392b-c4df-4a56-8bb4-75593ef65bb5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:55:25.000481 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.000376 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-service-ca" (OuterVolumeSpecName: "service-ca") pod "f12e392b-c4df-4a56-8bb4-75593ef65bb5" (UID: "f12e392b-c4df-4a56-8bb4-75593ef65bb5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:55:25.000529 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.000482 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-service-ca\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:55:25.000529 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.000476 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f12e392b-c4df-4a56-8bb4-75593ef65bb5" (UID: "f12e392b-c4df-4a56-8bb4-75593ef65bb5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:55:25.000529 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.000498 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-config\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:55:25.000529 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.000512 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-oauth-serving-cert\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:55:25.002264 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.002234 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12e392b-c4df-4a56-8bb4-75593ef65bb5-kube-api-access-mdq8b" (OuterVolumeSpecName: "kube-api-access-mdq8b") pod "f12e392b-c4df-4a56-8bb4-75593ef65bb5" (UID: "f12e392b-c4df-4a56-8bb4-75593ef65bb5"). InnerVolumeSpecName "kube-api-access-mdq8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:55:25.002384 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.002313 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f12e392b-c4df-4a56-8bb4-75593ef65bb5" (UID: "f12e392b-c4df-4a56-8bb4-75593ef65bb5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:55:25.002384 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.002339 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f12e392b-c4df-4a56-8bb4-75593ef65bb5" (UID: "f12e392b-c4df-4a56-8bb4-75593ef65bb5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:55:25.101661 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.101629 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12e392b-c4df-4a56-8bb4-75593ef65bb5-trusted-ca-bundle\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:55:25.101661 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.101659 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdq8b\" (UniqueName: \"kubernetes.io/projected/f12e392b-c4df-4a56-8bb4-75593ef65bb5-kube-api-access-mdq8b\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:55:25.101661 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.101669 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-oauth-config\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:55:25.102102 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.101677 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f12e392b-c4df-4a56-8bb4-75593ef65bb5-console-serving-cert\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:55:25.173793 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.173758 2569 generic.go:356] "Generic (PLEG): container finished" podID="f12e392b-c4df-4a56-8bb4-75593ef65bb5" containerID="b4e0676f371f69909918297e674e82c9c25b50128b0246caf24a6e14cc8ce013" exitCode=0 Jul 09 18:55:25.173972 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.173811 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4b9c584f-g2s7b" Jul 09 18:55:25.173972 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.173843 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4b9c584f-g2s7b" event={"ID":"f12e392b-c4df-4a56-8bb4-75593ef65bb5","Type":"ContainerDied","Data":"b4e0676f371f69909918297e674e82c9c25b50128b0246caf24a6e14cc8ce013"} Jul 09 18:55:25.173972 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.173905 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4b9c584f-g2s7b" event={"ID":"f12e392b-c4df-4a56-8bb4-75593ef65bb5","Type":"ContainerDied","Data":"307a804f3b62aefa126b12961b20b3f27d48fa09961a0e941329ede1b685487e"} Jul 09 18:55:25.173972 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.173938 2569 scope.go:117] "RemoveContainer" containerID="b4e0676f371f69909918297e674e82c9c25b50128b0246caf24a6e14cc8ce013" Jul 09 18:55:25.184099 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.184082 2569 scope.go:117] "RemoveContainer" containerID="b4e0676f371f69909918297e674e82c9c25b50128b0246caf24a6e14cc8ce013" Jul 09 18:55:25.184397 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:55:25.184377 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e0676f371f69909918297e674e82c9c25b50128b0246caf24a6e14cc8ce013\": container with ID starting with b4e0676f371f69909918297e674e82c9c25b50128b0246caf24a6e14cc8ce013 not found: ID does not exist" containerID="b4e0676f371f69909918297e674e82c9c25b50128b0246caf24a6e14cc8ce013" Jul 09 18:55:25.184455 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.184405 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e0676f371f69909918297e674e82c9c25b50128b0246caf24a6e14cc8ce013"} err="failed to get container status \"b4e0676f371f69909918297e674e82c9c25b50128b0246caf24a6e14cc8ce013\": rpc error: code = NotFound desc = could not find container \"b4e0676f371f69909918297e674e82c9c25b50128b0246caf24a6e14cc8ce013\": container with ID starting with b4e0676f371f69909918297e674e82c9c25b50128b0246caf24a6e14cc8ce013 not found: ID does not exist" Jul 09 18:55:25.209607 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.209569 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d4b9c584f-g2s7b"] Jul 09 18:55:25.223563 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.222034 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d4b9c584f-g2s7b"] Jul 09 18:55:25.694197 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:55:25.694167 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f12e392b-c4df-4a56-8bb4-75593ef65bb5" path="/var/lib/kubelet/pods/f12e392b-c4df-4a56-8bb4-75593ef65bb5/volumes" Jul 09 18:56:07.418438 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:07.418403 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 18:56:07.418963 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:07.418671 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="alertmanager" containerID="cri-o://c256d913686501087f45ce7eb0578e02735dc72403b3b7ccbcab5d603f6ea55b" gracePeriod=120 Jul 09 18:56:07.418963 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:07.418743 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="kube-rbac-proxy-metric" containerID="cri-o://c6742370d24c11d07efba7470dff902f20c274c19ef1e818dce75f6da7a30876" gracePeriod=120 Jul 09 18:56:07.418963 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:07.418798 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="prom-label-proxy" containerID="cri-o://f719066d12070dd96ddd128d5db091af271a36f36ae57b80fcc6cc71edfe4443" gracePeriod=120 Jul 09 18:56:07.418963 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:07.418817 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="config-reloader" containerID="cri-o://e6aebd81828216a8f2a792a40c45a56a31767244c276494aa31221abbcef53d9" gracePeriod=120 Jul 09 18:56:07.418963 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:07.418820 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="kube-rbac-proxy" containerID="cri-o://5703e42bcd8aeeea740218130c14835f6a9c15c1572844080e22657c4abbd74a" gracePeriod=120 Jul 09 18:56:07.418963 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:07.418760 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="kube-rbac-proxy-web" containerID="cri-o://ef5d0bec8c132ee5526ad9138e8b901d117fc51f916ca7ba83f82312cd1dd369" gracePeriod=120 Jul 09 18:56:08.333252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.333220 2569 generic.go:356] "Generic (PLEG): container finished" podID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerID="f719066d12070dd96ddd128d5db091af271a36f36ae57b80fcc6cc71edfe4443" exitCode=0 Jul 09 18:56:08.333252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.333247 2569 generic.go:356] "Generic (PLEG): container finished" podID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerID="c6742370d24c11d07efba7470dff902f20c274c19ef1e818dce75f6da7a30876" exitCode=0 Jul 09 18:56:08.333252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.333254 2569 generic.go:356] "Generic (PLEG): container finished" podID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerID="5703e42bcd8aeeea740218130c14835f6a9c15c1572844080e22657c4abbd74a" exitCode=0 Jul 09 18:56:08.333252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.333260 2569 generic.go:356] "Generic (PLEG): container finished" podID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerID="e6aebd81828216a8f2a792a40c45a56a31767244c276494aa31221abbcef53d9" exitCode=0 Jul 09 18:56:08.333252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.333267 2569 generic.go:356] "Generic (PLEG): container finished" podID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerID="c256d913686501087f45ce7eb0578e02735dc72403b3b7ccbcab5d603f6ea55b" exitCode=0 Jul 09 18:56:08.333546 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.333295 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cd93c96-1509-4341-bac4-67f67b8ad24d","Type":"ContainerDied","Data":"f719066d12070dd96ddd128d5db091af271a36f36ae57b80fcc6cc71edfe4443"} Jul 09 18:56:08.333546 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.333359 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cd93c96-1509-4341-bac4-67f67b8ad24d","Type":"ContainerDied","Data":"c6742370d24c11d07efba7470dff902f20c274c19ef1e818dce75f6da7a30876"} Jul 09 18:56:08.333546 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.333386 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cd93c96-1509-4341-bac4-67f67b8ad24d","Type":"ContainerDied","Data":"5703e42bcd8aeeea740218130c14835f6a9c15c1572844080e22657c4abbd74a"} Jul 09 18:56:08.333546 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.333410 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cd93c96-1509-4341-bac4-67f67b8ad24d","Type":"ContainerDied","Data":"e6aebd81828216a8f2a792a40c45a56a31767244c276494aa31221abbcef53d9"} Jul 09 18:56:08.333546 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.333433 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cd93c96-1509-4341-bac4-67f67b8ad24d","Type":"ContainerDied","Data":"c256d913686501087f45ce7eb0578e02735dc72403b3b7ccbcab5d603f6ea55b"} Jul 09 18:56:08.656315 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.656294 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:08.750603 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.750573 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7cd93c96-1509-4341-bac4-67f67b8ad24d-alertmanager-main-db\") pod \"7cd93c96-1509-4341-bac4-67f67b8ad24d\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " Jul 09 18:56:08.750781 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.750610 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7cd93c96-1509-4341-bac4-67f67b8ad24d-config-out\") pod \"7cd93c96-1509-4341-bac4-67f67b8ad24d\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " Jul 09 18:56:08.750781 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.750634 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7cd93c96-1509-4341-bac4-67f67b8ad24d\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " Jul 09 18:56:08.750781 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.750653 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cd93c96-1509-4341-bac4-67f67b8ad24d-alertmanager-trusted-ca-bundle\") pod \"7cd93c96-1509-4341-bac4-67f67b8ad24d\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " Jul 09 18:56:08.750781 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.750669 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxb44\" (UniqueName: \"kubernetes.io/projected/7cd93c96-1509-4341-bac4-67f67b8ad24d-kube-api-access-sxb44\") pod \"7cd93c96-1509-4341-bac4-67f67b8ad24d\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " Jul 09 18:56:08.751021 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.750805 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-web-config\") pod \"7cd93c96-1509-4341-bac4-67f67b8ad24d\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " Jul 09 18:56:08.751021 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.750841 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-main-tls\") pod \"7cd93c96-1509-4341-bac4-67f67b8ad24d\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " Jul 09 18:56:08.751021 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.750875 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-config-volume\") pod \"7cd93c96-1509-4341-bac4-67f67b8ad24d\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " Jul 09 18:56:08.751021 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.750913 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-cluster-tls-config\") pod \"7cd93c96-1509-4341-bac4-67f67b8ad24d\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " Jul 09 18:56:08.751021 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.750942 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cd93c96-1509-4341-bac4-67f67b8ad24d-metrics-client-ca\") pod \"7cd93c96-1509-4341-bac4-67f67b8ad24d\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " Jul 09 18:56:08.751021 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.750986 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7cd93c96-1509-4341-bac4-67f67b8ad24d-tls-assets\") pod \"7cd93c96-1509-4341-bac4-67f67b8ad24d\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " Jul 09 18:56:08.751021 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.750982 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd93c96-1509-4341-bac4-67f67b8ad24d-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "7cd93c96-1509-4341-bac4-67f67b8ad24d" (UID: "7cd93c96-1509-4341-bac4-67f67b8ad24d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:56:08.751021 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.751021 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy-web\") pod \"7cd93c96-1509-4341-bac4-67f67b8ad24d\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " Jul 09 18:56:08.751419 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.751073 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd93c96-1509-4341-bac4-67f67b8ad24d-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "7cd93c96-1509-4341-bac4-67f67b8ad24d" (UID: "7cd93c96-1509-4341-bac4-67f67b8ad24d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:56:08.751419 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.751080 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy\") pod \"7cd93c96-1509-4341-bac4-67f67b8ad24d\" (UID: \"7cd93c96-1509-4341-bac4-67f67b8ad24d\") " Jul 09 18:56:08.751419 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.751327 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cd93c96-1509-4341-bac4-67f67b8ad24d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:08.751419 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.751347 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7cd93c96-1509-4341-bac4-67f67b8ad24d-alertmanager-main-db\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:08.753525 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.753488 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd93c96-1509-4341-bac4-67f67b8ad24d-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "7cd93c96-1509-4341-bac4-67f67b8ad24d" (UID: "7cd93c96-1509-4341-bac4-67f67b8ad24d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:56:08.754085 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.753934 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd93c96-1509-4341-bac4-67f67b8ad24d-config-out" (OuterVolumeSpecName: "config-out") pod "7cd93c96-1509-4341-bac4-67f67b8ad24d" (UID: "7cd93c96-1509-4341-bac4-67f67b8ad24d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:56:08.754085 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.754073 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "7cd93c96-1509-4341-bac4-67f67b8ad24d" (UID: "7cd93c96-1509-4341-bac4-67f67b8ad24d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:08.754260 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.754130 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "7cd93c96-1509-4341-bac4-67f67b8ad24d" (UID: "7cd93c96-1509-4341-bac4-67f67b8ad24d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:08.754328 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.754302 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd93c96-1509-4341-bac4-67f67b8ad24d-kube-api-access-sxb44" (OuterVolumeSpecName: "kube-api-access-sxb44") pod "7cd93c96-1509-4341-bac4-67f67b8ad24d" (UID: "7cd93c96-1509-4341-bac4-67f67b8ad24d"). InnerVolumeSpecName "kube-api-access-sxb44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:56:08.754449 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.754426 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "7cd93c96-1509-4341-bac4-67f67b8ad24d" (UID: "7cd93c96-1509-4341-bac4-67f67b8ad24d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:08.754641 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.754617 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd93c96-1509-4341-bac4-67f67b8ad24d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7cd93c96-1509-4341-bac4-67f67b8ad24d" (UID: "7cd93c96-1509-4341-bac4-67f67b8ad24d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:56:08.755749 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.755728 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-config-volume" (OuterVolumeSpecName: "config-volume") pod "7cd93c96-1509-4341-bac4-67f67b8ad24d" (UID: "7cd93c96-1509-4341-bac4-67f67b8ad24d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:08.755823 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.755795 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "7cd93c96-1509-4341-bac4-67f67b8ad24d" (UID: "7cd93c96-1509-4341-bac4-67f67b8ad24d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:08.758537 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.758464 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "7cd93c96-1509-4341-bac4-67f67b8ad24d" (UID: "7cd93c96-1509-4341-bac4-67f67b8ad24d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:08.763028 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.763005 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-web-config" (OuterVolumeSpecName: "web-config") pod "7cd93c96-1509-4341-bac4-67f67b8ad24d" (UID: "7cd93c96-1509-4341-bac4-67f67b8ad24d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:08.852048 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.851968 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:08.852048 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.851999 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:08.852048 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.852012 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7cd93c96-1509-4341-bac4-67f67b8ad24d-config-out\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:08.852048 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.852024 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:08.852048 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.852038 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxb44\" (UniqueName: \"kubernetes.io/projected/7cd93c96-1509-4341-bac4-67f67b8ad24d-kube-api-access-sxb44\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:08.852048 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.852051 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-web-config\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:08.852322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.852063 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-secret-alertmanager-main-tls\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:08.852322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.852075 2569 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-config-volume\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:08.852322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.852085 2569 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7cd93c96-1509-4341-bac4-67f67b8ad24d-cluster-tls-config\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:08.852322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.852096 2569 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cd93c96-1509-4341-bac4-67f67b8ad24d-metrics-client-ca\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:08.852322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:08.852107 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7cd93c96-1509-4341-bac4-67f67b8ad24d-tls-assets\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:09.339711 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.339673 2569 generic.go:356] "Generic (PLEG): container finished" podID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerID="ef5d0bec8c132ee5526ad9138e8b901d117fc51f916ca7ba83f82312cd1dd369" exitCode=0 Jul 09 18:56:09.339896 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.339752 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cd93c96-1509-4341-bac4-67f67b8ad24d","Type":"ContainerDied","Data":"ef5d0bec8c132ee5526ad9138e8b901d117fc51f916ca7ba83f82312cd1dd369"} Jul 09 18:56:09.339896 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.339763 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.339896 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.339853 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7cd93c96-1509-4341-bac4-67f67b8ad24d","Type":"ContainerDied","Data":"2d12ddda1dd329b038b1c538ad67b49a05da28186e90ea571af2801b3a15248f"} Jul 09 18:56:09.339896 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.339891 2569 scope.go:117] "RemoveContainer" containerID="f719066d12070dd96ddd128d5db091af271a36f36ae57b80fcc6cc71edfe4443" Jul 09 18:56:09.347640 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.347616 2569 scope.go:117] "RemoveContainer" containerID="c6742370d24c11d07efba7470dff902f20c274c19ef1e818dce75f6da7a30876" Jul 09 18:56:09.354936 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.354917 2569 scope.go:117] "RemoveContainer" containerID="5703e42bcd8aeeea740218130c14835f6a9c15c1572844080e22657c4abbd74a" Jul 09 18:56:09.360957 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.360922 2569 scope.go:117] "RemoveContainer" containerID="ef5d0bec8c132ee5526ad9138e8b901d117fc51f916ca7ba83f82312cd1dd369" Jul 09 18:56:09.365656 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.365635 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 18:56:09.370814 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.370794 2569 scope.go:117] "RemoveContainer" containerID="e6aebd81828216a8f2a792a40c45a56a31767244c276494aa31221abbcef53d9" Jul 09 18:56:09.370887 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.370871 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 18:56:09.377463 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.377446 2569 scope.go:117] "RemoveContainer" containerID="c256d913686501087f45ce7eb0578e02735dc72403b3b7ccbcab5d603f6ea55b" Jul 09 18:56:09.384125 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.384107 2569 scope.go:117] "RemoveContainer" containerID="42bd535575dc57b2c1ff5ac62536d0862ba4c7a8d4e89cb6aecc81cfcd8cdb86" Jul 09 18:56:09.390601 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.390582 2569 scope.go:117] "RemoveContainer" containerID="f719066d12070dd96ddd128d5db091af271a36f36ae57b80fcc6cc71edfe4443" Jul 09 18:56:09.390864 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:56:09.390847 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f719066d12070dd96ddd128d5db091af271a36f36ae57b80fcc6cc71edfe4443\": container with ID starting with f719066d12070dd96ddd128d5db091af271a36f36ae57b80fcc6cc71edfe4443 not found: ID does not exist" containerID="f719066d12070dd96ddd128d5db091af271a36f36ae57b80fcc6cc71edfe4443" Jul 09 18:56:09.390915 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.390872 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f719066d12070dd96ddd128d5db091af271a36f36ae57b80fcc6cc71edfe4443"} err="failed to get container status \"f719066d12070dd96ddd128d5db091af271a36f36ae57b80fcc6cc71edfe4443\": rpc error: code = NotFound desc = could not find container \"f719066d12070dd96ddd128d5db091af271a36f36ae57b80fcc6cc71edfe4443\": container with ID starting with f719066d12070dd96ddd128d5db091af271a36f36ae57b80fcc6cc71edfe4443 not found: ID does not exist" Jul 09 18:56:09.390915 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.390891 2569 scope.go:117] "RemoveContainer" containerID="c6742370d24c11d07efba7470dff902f20c274c19ef1e818dce75f6da7a30876" Jul 09 18:56:09.391136 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:56:09.391117 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6742370d24c11d07efba7470dff902f20c274c19ef1e818dce75f6da7a30876\": container with ID starting with c6742370d24c11d07efba7470dff902f20c274c19ef1e818dce75f6da7a30876 not found: ID does not exist" containerID="c6742370d24c11d07efba7470dff902f20c274c19ef1e818dce75f6da7a30876" Jul 09 18:56:09.391181 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.391141 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6742370d24c11d07efba7470dff902f20c274c19ef1e818dce75f6da7a30876"} err="failed to get container status \"c6742370d24c11d07efba7470dff902f20c274c19ef1e818dce75f6da7a30876\": rpc error: code = NotFound desc = could not find container \"c6742370d24c11d07efba7470dff902f20c274c19ef1e818dce75f6da7a30876\": container with ID starting with c6742370d24c11d07efba7470dff902f20c274c19ef1e818dce75f6da7a30876 not found: ID does not exist" Jul 09 18:56:09.391181 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.391157 2569 scope.go:117] "RemoveContainer" containerID="5703e42bcd8aeeea740218130c14835f6a9c15c1572844080e22657c4abbd74a" Jul 09 18:56:09.391364 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:56:09.391347 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5703e42bcd8aeeea740218130c14835f6a9c15c1572844080e22657c4abbd74a\": container with ID starting with 5703e42bcd8aeeea740218130c14835f6a9c15c1572844080e22657c4abbd74a not found: ID does not exist" containerID="5703e42bcd8aeeea740218130c14835f6a9c15c1572844080e22657c4abbd74a" Jul 09 18:56:09.391407 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.391370 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5703e42bcd8aeeea740218130c14835f6a9c15c1572844080e22657c4abbd74a"} err="failed to get container status \"5703e42bcd8aeeea740218130c14835f6a9c15c1572844080e22657c4abbd74a\": rpc error: code = NotFound desc = could not find container \"5703e42bcd8aeeea740218130c14835f6a9c15c1572844080e22657c4abbd74a\": container with ID starting with 5703e42bcd8aeeea740218130c14835f6a9c15c1572844080e22657c4abbd74a not found: ID does not exist" Jul 09 18:56:09.391407 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.391388 2569 scope.go:117] "RemoveContainer" containerID="ef5d0bec8c132ee5526ad9138e8b901d117fc51f916ca7ba83f82312cd1dd369" Jul 09 18:56:09.391591 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:56:09.391575 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef5d0bec8c132ee5526ad9138e8b901d117fc51f916ca7ba83f82312cd1dd369\": container with ID starting with ef5d0bec8c132ee5526ad9138e8b901d117fc51f916ca7ba83f82312cd1dd369 not found: ID does not exist" containerID="ef5d0bec8c132ee5526ad9138e8b901d117fc51f916ca7ba83f82312cd1dd369" Jul 09 18:56:09.391630 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.391596 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef5d0bec8c132ee5526ad9138e8b901d117fc51f916ca7ba83f82312cd1dd369"} err="failed to get container status \"ef5d0bec8c132ee5526ad9138e8b901d117fc51f916ca7ba83f82312cd1dd369\": rpc error: code = NotFound desc = could not find container \"ef5d0bec8c132ee5526ad9138e8b901d117fc51f916ca7ba83f82312cd1dd369\": container with ID starting with ef5d0bec8c132ee5526ad9138e8b901d117fc51f916ca7ba83f82312cd1dd369 not found: ID does not exist" Jul 09 18:56:09.391630 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.391609 2569 scope.go:117] "RemoveContainer" containerID="e6aebd81828216a8f2a792a40c45a56a31767244c276494aa31221abbcef53d9" Jul 09 18:56:09.391842 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:56:09.391827 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6aebd81828216a8f2a792a40c45a56a31767244c276494aa31221abbcef53d9\": container with ID starting with e6aebd81828216a8f2a792a40c45a56a31767244c276494aa31221abbcef53d9 not found: ID does not exist" containerID="e6aebd81828216a8f2a792a40c45a56a31767244c276494aa31221abbcef53d9" Jul 09 18:56:09.391886 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.391843 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6aebd81828216a8f2a792a40c45a56a31767244c276494aa31221abbcef53d9"} err="failed to get container status \"e6aebd81828216a8f2a792a40c45a56a31767244c276494aa31221abbcef53d9\": rpc error: code = NotFound desc = could not find container \"e6aebd81828216a8f2a792a40c45a56a31767244c276494aa31221abbcef53d9\": container with ID starting with e6aebd81828216a8f2a792a40c45a56a31767244c276494aa31221abbcef53d9 not found: ID does not exist" Jul 09 18:56:09.391886 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.391857 2569 scope.go:117] "RemoveContainer" containerID="c256d913686501087f45ce7eb0578e02735dc72403b3b7ccbcab5d603f6ea55b" Jul 09 18:56:09.392094 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:56:09.392075 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c256d913686501087f45ce7eb0578e02735dc72403b3b7ccbcab5d603f6ea55b\": container with ID starting with c256d913686501087f45ce7eb0578e02735dc72403b3b7ccbcab5d603f6ea55b not found: ID does not exist" containerID="c256d913686501087f45ce7eb0578e02735dc72403b3b7ccbcab5d603f6ea55b" Jul 09 18:56:09.392142 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.392099 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c256d913686501087f45ce7eb0578e02735dc72403b3b7ccbcab5d603f6ea55b"} err="failed to get container status \"c256d913686501087f45ce7eb0578e02735dc72403b3b7ccbcab5d603f6ea55b\": rpc error: code = NotFound desc = could not find container \"c256d913686501087f45ce7eb0578e02735dc72403b3b7ccbcab5d603f6ea55b\": container with ID starting with c256d913686501087f45ce7eb0578e02735dc72403b3b7ccbcab5d603f6ea55b not found: ID does not exist" Jul 09 18:56:09.392142 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.392114 2569 scope.go:117] "RemoveContainer" containerID="42bd535575dc57b2c1ff5ac62536d0862ba4c7a8d4e89cb6aecc81cfcd8cdb86" Jul 09 18:56:09.392337 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:56:09.392322 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42bd535575dc57b2c1ff5ac62536d0862ba4c7a8d4e89cb6aecc81cfcd8cdb86\": container with ID starting with 42bd535575dc57b2c1ff5ac62536d0862ba4c7a8d4e89cb6aecc81cfcd8cdb86 not found: ID does not exist" containerID="42bd535575dc57b2c1ff5ac62536d0862ba4c7a8d4e89cb6aecc81cfcd8cdb86" Jul 09 18:56:09.392375 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.392339 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42bd535575dc57b2c1ff5ac62536d0862ba4c7a8d4e89cb6aecc81cfcd8cdb86"} err="failed to get container status \"42bd535575dc57b2c1ff5ac62536d0862ba4c7a8d4e89cb6aecc81cfcd8cdb86\": rpc error: code = NotFound desc = could not find container \"42bd535575dc57b2c1ff5ac62536d0862ba4c7a8d4e89cb6aecc81cfcd8cdb86\": container with ID starting with 42bd535575dc57b2c1ff5ac62536d0862ba4c7a8d4e89cb6aecc81cfcd8cdb86 not found: ID does not exist" Jul 09 18:56:09.397641 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.397615 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 18:56:09.397893 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.397878 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="init-config-reloader" Jul 09 18:56:09.397985 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.397897 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="init-config-reloader" Jul 09 18:56:09.397985 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.397910 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="kube-rbac-proxy-metric" Jul 09 18:56:09.397985 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.397920 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="kube-rbac-proxy-metric" Jul 09 18:56:09.397985 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.397931 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f12e392b-c4df-4a56-8bb4-75593ef65bb5" containerName="console" Jul 09 18:56:09.397985 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.397940 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12e392b-c4df-4a56-8bb4-75593ef65bb5" containerName="console" Jul 09 18:56:09.397985 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.397974 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="alertmanager" Jul 09 18:56:09.397985 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.397983 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="alertmanager" Jul 09 18:56:09.398335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.397991 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="prom-label-proxy" Jul 09 18:56:09.398335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.397999 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="prom-label-proxy" Jul 09 18:56:09.398335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.398022 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="config-reloader" Jul 09 18:56:09.398335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.398030 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="config-reloader" Jul 09 18:56:09.398335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.398041 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="kube-rbac-proxy" Jul 09 18:56:09.398335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.398050 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="kube-rbac-proxy" Jul 09 18:56:09.398335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.398060 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="kube-rbac-proxy-web" Jul 09 18:56:09.398335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.398068 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="kube-rbac-proxy-web" Jul 09 18:56:09.398335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.398135 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f12e392b-c4df-4a56-8bb4-75593ef65bb5" containerName="console" Jul 09 18:56:09.398335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.398148 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="kube-rbac-proxy" Jul 09 18:56:09.398335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.398158 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="alertmanager" Jul 09 18:56:09.398335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.398166 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="config-reloader" Jul 09 18:56:09.398335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.398175 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="kube-rbac-proxy-web" Jul 09 18:56:09.398335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.398183 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="kube-rbac-proxy-metric" Jul 09 18:56:09.398335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.398191 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" containerName="prom-label-proxy" Jul 09 18:56:09.402066 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.402049 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.407693 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.407672 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 09 18:56:09.408344 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.407696 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 09 18:56:09.408344 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.407809 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 09 18:56:09.408344 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.408146 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5ddg2\"" Jul 09 18:56:09.408344 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.408280 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 09 18:56:09.408691 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.408359 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 09 18:56:09.408691 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.408368 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 09 18:56:09.408691 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.408554 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 09 18:56:09.408888 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.408876 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 09 18:56:09.415407 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.415389 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 09 18:56:09.421555 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.421534 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 18:56:09.457503 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.457468 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.457503 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.457503 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-config-volume\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.457716 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.457532 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59b649d0-cbd9-4828-a8b3-13fa14188249-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.457716 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.457601 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-web-config\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.457716 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.457672 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59b649d0-cbd9-4828-a8b3-13fa14188249-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.457859 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.457709 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b649d0-cbd9-4828-a8b3-13fa14188249-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.457859 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.457746 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.457859 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.457807 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.457859 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.457840 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.458059 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.457865 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.458059 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.457892 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59b649d0-cbd9-4828-a8b3-13fa14188249-config-out\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.458059 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.457922 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59b649d0-cbd9-4828-a8b3-13fa14188249-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.458059 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.457965 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d49rf\" (UniqueName: \"kubernetes.io/projected/59b649d0-cbd9-4828-a8b3-13fa14188249-kube-api-access-d49rf\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.559196 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.559158 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.559196 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.559200 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.559447 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.559219 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.559447 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.559236 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59b649d0-cbd9-4828-a8b3-13fa14188249-config-out\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.559447 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.559261 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59b649d0-cbd9-4828-a8b3-13fa14188249-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.559447 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.559284 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49rf\" (UniqueName: \"kubernetes.io/projected/59b649d0-cbd9-4828-a8b3-13fa14188249-kube-api-access-d49rf\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.559447 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.559317 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.559447 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.559346 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-config-volume\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.559447 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.559373 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59b649d0-cbd9-4828-a8b3-13fa14188249-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.559447 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.559407 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-web-config\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.559447 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.559440 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59b649d0-cbd9-4828-a8b3-13fa14188249-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.559854 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.559466 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b649d0-cbd9-4828-a8b3-13fa14188249-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.559854 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.559495 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.562414 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.562282 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.562414 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.562345 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-web-config\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.562414 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.562379 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59b649d0-cbd9-4828-a8b3-13fa14188249-config-out\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.562414 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.562410 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.562681 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.562496 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.562681 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.562586 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.562787 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.562716 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.562787 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.562775 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59b649d0-cbd9-4828-a8b3-13fa14188249-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.563019 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.563001 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59b649d0-cbd9-4828-a8b3-13fa14188249-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.563380 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.563357 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b649d0-cbd9-4828-a8b3-13fa14188249-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.564057 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.564034 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59b649d0-cbd9-4828-a8b3-13fa14188249-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.564335 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.564319 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59b649d0-cbd9-4828-a8b3-13fa14188249-config-volume\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.568069 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.568051 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49rf\" (UniqueName: \"kubernetes.io/projected/59b649d0-cbd9-4828-a8b3-13fa14188249-kube-api-access-d49rf\") pod \"alertmanager-main-0\" (UID: \"59b649d0-cbd9-4828-a8b3-13fa14188249\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.693581 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.693502 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cd93c96-1509-4341-bac4-67f67b8ad24d" path="/var/lib/kubelet/pods/7cd93c96-1509-4341-bac4-67f67b8ad24d/volumes" Jul 09 18:56:09.711308 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.711282 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 18:56:09.845519 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:09.845489 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 18:56:09.849169 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:56:09.849140 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b649d0_cbd9_4828_a8b3_13fa14188249.slice/crio-2452e6481faadee0f295cccc917787d51a7b7ea9f173fc6712d2b582bee01c56 WatchSource:0}: Error finding container 2452e6481faadee0f295cccc917787d51a7b7ea9f173fc6712d2b582bee01c56: Status 404 returned error can't find the container with id 2452e6481faadee0f295cccc917787d51a7b7ea9f173fc6712d2b582bee01c56 Jul 09 18:56:10.344695 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:10.344660 2569 generic.go:356] "Generic (PLEG): container finished" podID="59b649d0-cbd9-4828-a8b3-13fa14188249" containerID="0e5030c8f3d5e59676f8268d80fd5eeb356448a801bfa5c6625798ef0b616eea" exitCode=0 Jul 09 18:56:10.344859 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:10.344743 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59b649d0-cbd9-4828-a8b3-13fa14188249","Type":"ContainerDied","Data":"0e5030c8f3d5e59676f8268d80fd5eeb356448a801bfa5c6625798ef0b616eea"} Jul 09 18:56:10.344859 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:10.344811 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59b649d0-cbd9-4828-a8b3-13fa14188249","Type":"ContainerStarted","Data":"2452e6481faadee0f295cccc917787d51a7b7ea9f173fc6712d2b582bee01c56"} Jul 09 18:56:11.351928 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:11.351895 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59b649d0-cbd9-4828-a8b3-13fa14188249","Type":"ContainerStarted","Data":"99540b2b1d200b7dcb53cbdf886dfa9404bafb203a5e3ac51b0ace54442153d9"} Jul 09 18:56:11.352320 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:11.351988 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59b649d0-cbd9-4828-a8b3-13fa14188249","Type":"ContainerStarted","Data":"8cd3d28235caebb42a703d73fe702d23f39beb86eb1fca16ce1dbe35c49f254e"} Jul 09 18:56:11.352320 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:11.352017 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59b649d0-cbd9-4828-a8b3-13fa14188249","Type":"ContainerStarted","Data":"b32b70a24320cfa90a0624d3468efae81630aede86f2bdff538d488808aae02a"} Jul 09 18:56:11.352320 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:11.352041 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59b649d0-cbd9-4828-a8b3-13fa14188249","Type":"ContainerStarted","Data":"a7423c0cbcd0997c9a7e003129e783f231ce4721b46a26b608008d08943ce4d2"} Jul 09 18:56:11.352320 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:11.352065 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59b649d0-cbd9-4828-a8b3-13fa14188249","Type":"ContainerStarted","Data":"e85b1fbd32856effeeb7a626c60c3f60dfaa412d7e0de52737751c99a4cfc7bd"} Jul 09 18:56:11.352320 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:11.352090 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59b649d0-cbd9-4828-a8b3-13fa14188249","Type":"ContainerStarted","Data":"e8e43dfdac1c3f9a3e5e27bb5ec9a5cf1cc544e6bd36490e6acca232b91261ac"} Jul 09 18:56:11.380235 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:11.380128 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.363981532 podStartE2EDuration="2.380109945s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.128413ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:56:11.378663283 +0000 UTC m=+156.203826320" watchObservedRunningTime="2026-07-09 18:56:11.380109945 +0000 UTC m=+156.205272983" Jul 09 18:56:22.553480 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.553441 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d5567bcd7-hzlnv"] Jul 09 18:56:22.557073 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.557048 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.569104 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.569074 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5567bcd7-hzlnv"] Jul 09 18:56:22.663694 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.663657 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a54ec556-a025-4a3e-8fff-7de0cd201187-console-serving-cert\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.663694 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.663693 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a54ec556-a025-4a3e-8fff-7de0cd201187-console-oauth-config\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.663897 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.663714 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-686k9\" (UniqueName: \"kubernetes.io/projected/a54ec556-a025-4a3e-8fff-7de0cd201187-kube-api-access-686k9\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.663897 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.663744 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-service-ca\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.663897 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.663761 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-oauth-serving-cert\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.663897 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.663777 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-trusted-ca-bundle\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.663897 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.663811 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-console-config\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.764436 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.764401 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-console-config\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.764524 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.764449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a54ec556-a025-4a3e-8fff-7de0cd201187-console-serving-cert\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.764524 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.764470 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a54ec556-a025-4a3e-8fff-7de0cd201187-console-oauth-config\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.764524 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.764504 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-686k9\" (UniqueName: \"kubernetes.io/projected/a54ec556-a025-4a3e-8fff-7de0cd201187-kube-api-access-686k9\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.764649 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.764554 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-service-ca\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.764649 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.764576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-oauth-serving-cert\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.764649 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.764598 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-trusted-ca-bundle\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.765287 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.765261 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-service-ca\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.765407 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.765302 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-oauth-serving-cert\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.765407 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.765278 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-console-config\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.765559 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.765535 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-trusted-ca-bundle\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.767133 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.767108 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a54ec556-a025-4a3e-8fff-7de0cd201187-console-oauth-config\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.767133 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.767119 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a54ec556-a025-4a3e-8fff-7de0cd201187-console-serving-cert\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.774180 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.774153 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-686k9\" (UniqueName: \"kubernetes.io/projected/a54ec556-a025-4a3e-8fff-7de0cd201187-kube-api-access-686k9\") pod \"console-5d5567bcd7-hzlnv\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.865548 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.865457 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:22.988739 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:22.988707 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5567bcd7-hzlnv"] Jul 09 18:56:22.991372 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:56:22.991348 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54ec556_a025_4a3e_8fff_7de0cd201187.slice/crio-0a9d028f5315ba589efa161feeb810d3b7efd9eb350f1f129f25191059a2619e WatchSource:0}: Error finding container 0a9d028f5315ba589efa161feeb810d3b7efd9eb350f1f129f25191059a2619e: Status 404 returned error can't find the container with id 0a9d028f5315ba589efa161feeb810d3b7efd9eb350f1f129f25191059a2619e Jul 09 18:56:23.399489 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:23.399450 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5567bcd7-hzlnv" event={"ID":"a54ec556-a025-4a3e-8fff-7de0cd201187","Type":"ContainerStarted","Data":"dbbd03cd93ae1daf4c13bfdb0eca709d2f9490a444fb70254ea576fb028db36f"} Jul 09 18:56:23.399659 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:23.399507 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5567bcd7-hzlnv" event={"ID":"a54ec556-a025-4a3e-8fff-7de0cd201187","Type":"ContainerStarted","Data":"0a9d028f5315ba589efa161feeb810d3b7efd9eb350f1f129f25191059a2619e"} Jul 09 18:56:23.418940 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:23.418884 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d5567bcd7-hzlnv" podStartSLOduration=1.418868434 podStartE2EDuration="1.418868434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:56:23.417601887 +0000 UTC m=+168.242764927" watchObservedRunningTime="2026-07-09 18:56:23.418868434 +0000 UTC m=+168.244031541" Jul 09 18:56:32.865813 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:32.865736 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:32.865813 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:32.865801 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:32.870586 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:32.870564 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:33.437072 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:33.437044 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 18:56:33.489110 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:33.489079 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75b5cc9556-x9gwz"] Jul 09 18:56:58.508562 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.508520 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75b5cc9556-x9gwz" podUID="ca815a25-029a-4db9-8254-1b1e891bfeeb" containerName="console" containerID="cri-o://dfe5ce2b97706be2beb7221a504c1c05d806d0d0ff4db1f3f449c287936a3595" gracePeriod=15 Jul 09 18:56:58.747396 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.747366 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:56:58.756328 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.756308 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-service-ca\") pod \"ca815a25-029a-4db9-8254-1b1e891bfeeb\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " Jul 09 18:56:58.756450 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.756421 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-config\") pod \"ca815a25-029a-4db9-8254-1b1e891bfeeb\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " Jul 09 18:56:58.756519 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.756463 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-oauth-serving-cert\") pod \"ca815a25-029a-4db9-8254-1b1e891bfeeb\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " Jul 09 18:56:58.756519 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.756499 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5726\" (UniqueName: \"kubernetes.io/projected/ca815a25-029a-4db9-8254-1b1e891bfeeb-kube-api-access-q5726\") pod \"ca815a25-029a-4db9-8254-1b1e891bfeeb\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " Jul 09 18:56:58.756619 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.756529 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-oauth-config\") pod \"ca815a25-029a-4db9-8254-1b1e891bfeeb\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " Jul 09 18:56:58.756619 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.756571 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-serving-cert\") pod \"ca815a25-029a-4db9-8254-1b1e891bfeeb\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " Jul 09 18:56:58.756619 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.756572 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-service-ca" (OuterVolumeSpecName: "service-ca") pod "ca815a25-029a-4db9-8254-1b1e891bfeeb" (UID: "ca815a25-029a-4db9-8254-1b1e891bfeeb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:56:58.756758 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.756639 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-trusted-ca-bundle\") pod \"ca815a25-029a-4db9-8254-1b1e891bfeeb\" (UID: \"ca815a25-029a-4db9-8254-1b1e891bfeeb\") " Jul 09 18:56:58.756868 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.756817 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-config" (OuterVolumeSpecName: "console-config") pod "ca815a25-029a-4db9-8254-1b1e891bfeeb" (UID: "ca815a25-029a-4db9-8254-1b1e891bfeeb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:56:58.756868 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.756839 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-service-ca\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:58.757032 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.756892 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ca815a25-029a-4db9-8254-1b1e891bfeeb" (UID: "ca815a25-029a-4db9-8254-1b1e891bfeeb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:56:58.757071 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.757031 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ca815a25-029a-4db9-8254-1b1e891bfeeb" (UID: "ca815a25-029a-4db9-8254-1b1e891bfeeb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:56:58.758662 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.758604 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca815a25-029a-4db9-8254-1b1e891bfeeb-kube-api-access-q5726" (OuterVolumeSpecName: "kube-api-access-q5726") pod "ca815a25-029a-4db9-8254-1b1e891bfeeb" (UID: "ca815a25-029a-4db9-8254-1b1e891bfeeb"). InnerVolumeSpecName "kube-api-access-q5726". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:56:58.758662 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.758631 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ca815a25-029a-4db9-8254-1b1e891bfeeb" (UID: "ca815a25-029a-4db9-8254-1b1e891bfeeb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:58.758790 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.758687 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ca815a25-029a-4db9-8254-1b1e891bfeeb" (UID: "ca815a25-029a-4db9-8254-1b1e891bfeeb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:58.857798 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.857765 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-serving-cert\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:58.857798 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.857796 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-trusted-ca-bundle\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:58.857798 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.857810 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-config\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:58.858082 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.857821 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca815a25-029a-4db9-8254-1b1e891bfeeb-oauth-serving-cert\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:58.858082 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.857835 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5726\" (UniqueName: \"kubernetes.io/projected/ca815a25-029a-4db9-8254-1b1e891bfeeb-kube-api-access-q5726\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:58.858082 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:58.857848 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca815a25-029a-4db9-8254-1b1e891bfeeb-console-oauth-config\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:56:59.525206 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:59.525172 2569 generic.go:356] "Generic (PLEG): container finished" podID="ca815a25-029a-4db9-8254-1b1e891bfeeb" containerID="dfe5ce2b97706be2beb7221a504c1c05d806d0d0ff4db1f3f449c287936a3595" exitCode=0 Jul 09 18:56:59.525635 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:59.525217 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75b5cc9556-x9gwz" Jul 09 18:56:59.525635 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:59.525258 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75b5cc9556-x9gwz" event={"ID":"ca815a25-029a-4db9-8254-1b1e891bfeeb","Type":"ContainerDied","Data":"dfe5ce2b97706be2beb7221a504c1c05d806d0d0ff4db1f3f449c287936a3595"} Jul 09 18:56:59.525635 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:59.525318 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75b5cc9556-x9gwz" event={"ID":"ca815a25-029a-4db9-8254-1b1e891bfeeb","Type":"ContainerDied","Data":"1bebd540f7401861dcd9e176f3cdb18e983ee42ae73d3e2be479d6ed376bea28"} Jul 09 18:56:59.525635 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:59.525349 2569 scope.go:117] "RemoveContainer" containerID="dfe5ce2b97706be2beb7221a504c1c05d806d0d0ff4db1f3f449c287936a3595" Jul 09 18:56:59.533529 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:59.533515 2569 scope.go:117] "RemoveContainer" containerID="dfe5ce2b97706be2beb7221a504c1c05d806d0d0ff4db1f3f449c287936a3595" Jul 09 18:56:59.533775 ip-10-0-133-185 kubenswrapper[2569]: E0709 18:56:59.533760 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe5ce2b97706be2beb7221a504c1c05d806d0d0ff4db1f3f449c287936a3595\": container with ID starting with dfe5ce2b97706be2beb7221a504c1c05d806d0d0ff4db1f3f449c287936a3595 not found: ID does not exist" containerID="dfe5ce2b97706be2beb7221a504c1c05d806d0d0ff4db1f3f449c287936a3595" Jul 09 18:56:59.533837 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:59.533787 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe5ce2b97706be2beb7221a504c1c05d806d0d0ff4db1f3f449c287936a3595"} err="failed to get container status \"dfe5ce2b97706be2beb7221a504c1c05d806d0d0ff4db1f3f449c287936a3595\": rpc error: code = NotFound desc = could not find container \"dfe5ce2b97706be2beb7221a504c1c05d806d0d0ff4db1f3f449c287936a3595\": container with ID starting with dfe5ce2b97706be2beb7221a504c1c05d806d0d0ff4db1f3f449c287936a3595 not found: ID does not exist" Jul 09 18:56:59.546823 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:59.546803 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75b5cc9556-x9gwz"] Jul 09 18:56:59.553785 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:59.553758 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75b5cc9556-x9gwz"] Jul 09 18:56:59.692865 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:56:59.692831 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca815a25-029a-4db9-8254-1b1e891bfeeb" path="/var/lib/kubelet/pods/ca815a25-029a-4db9-8254-1b1e891bfeeb/volumes" Jul 09 18:57:12.657261 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.657230 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl"] Jul 09 18:57:12.657622 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.657518 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca815a25-029a-4db9-8254-1b1e891bfeeb" containerName="console" Jul 09 18:57:12.657622 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.657534 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca815a25-029a-4db9-8254-1b1e891bfeeb" containerName="console" Jul 09 18:57:12.657622 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.657601 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca815a25-029a-4db9-8254-1b1e891bfeeb" containerName="console" Jul 09 18:57:12.662488 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.662453 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" Jul 09 18:57:12.667012 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.666986 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 18:57:12.667178 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.667166 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 18:57:12.667224 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.667162 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cd2fs\"" Jul 09 18:57:12.676902 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.676879 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl"] Jul 09 18:57:12.756429 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.756395 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5fa1eac-5061-4a8a-ae23-648aed9d3760-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl\" (UID: \"b5fa1eac-5061-4a8a-ae23-648aed9d3760\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" Jul 09 18:57:12.756619 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.756465 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh4r7\" (UniqueName: \"kubernetes.io/projected/b5fa1eac-5061-4a8a-ae23-648aed9d3760-kube-api-access-zh4r7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl\" (UID: \"b5fa1eac-5061-4a8a-ae23-648aed9d3760\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" Jul 09 18:57:12.756619 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.756543 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5fa1eac-5061-4a8a-ae23-648aed9d3760-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl\" (UID: \"b5fa1eac-5061-4a8a-ae23-648aed9d3760\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" Jul 09 18:57:12.857217 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.857190 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5fa1eac-5061-4a8a-ae23-648aed9d3760-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl\" (UID: \"b5fa1eac-5061-4a8a-ae23-648aed9d3760\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" Jul 09 18:57:12.857351 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.857250 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh4r7\" (UniqueName: \"kubernetes.io/projected/b5fa1eac-5061-4a8a-ae23-648aed9d3760-kube-api-access-zh4r7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl\" (UID: \"b5fa1eac-5061-4a8a-ae23-648aed9d3760\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" Jul 09 18:57:12.857351 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.857278 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5fa1eac-5061-4a8a-ae23-648aed9d3760-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl\" (UID: \"b5fa1eac-5061-4a8a-ae23-648aed9d3760\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" Jul 09 18:57:12.857643 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.857620 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5fa1eac-5061-4a8a-ae23-648aed9d3760-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl\" (UID: \"b5fa1eac-5061-4a8a-ae23-648aed9d3760\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" Jul 09 18:57:12.857706 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.857642 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5fa1eac-5061-4a8a-ae23-648aed9d3760-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl\" (UID: \"b5fa1eac-5061-4a8a-ae23-648aed9d3760\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" Jul 09 18:57:12.867461 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.867430 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh4r7\" (UniqueName: \"kubernetes.io/projected/b5fa1eac-5061-4a8a-ae23-648aed9d3760-kube-api-access-zh4r7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl\" (UID: \"b5fa1eac-5061-4a8a-ae23-648aed9d3760\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" Jul 09 18:57:12.971548 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:12.971519 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" Jul 09 18:57:13.096798 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:13.096767 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl"] Jul 09 18:57:13.100039 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:57:13.100012 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fa1eac_5061_4a8a_ae23_648aed9d3760.slice/crio-f7b359232937bc9d3b48db58bedac401ed7b371f406731bd76903d6f16c45b19 WatchSource:0}: Error finding container f7b359232937bc9d3b48db58bedac401ed7b371f406731bd76903d6f16c45b19: Status 404 returned error can't find the container with id f7b359232937bc9d3b48db58bedac401ed7b371f406731bd76903d6f16c45b19 Jul 09 18:57:13.577732 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:13.577700 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" event={"ID":"b5fa1eac-5061-4a8a-ae23-648aed9d3760","Type":"ContainerStarted","Data":"f7b359232937bc9d3b48db58bedac401ed7b371f406731bd76903d6f16c45b19"} Jul 09 18:57:18.597784 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:18.597753 2569 generic.go:356] "Generic (PLEG): container finished" podID="b5fa1eac-5061-4a8a-ae23-648aed9d3760" containerID="5bb4de5a942a57506d453dc5396918306315480b9a030358de393c870ee80aa2" exitCode=0 Jul 09 18:57:18.598192 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:18.597791 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" event={"ID":"b5fa1eac-5061-4a8a-ae23-648aed9d3760","Type":"ContainerDied","Data":"5bb4de5a942a57506d453dc5396918306315480b9a030358de393c870ee80aa2"} Jul 09 18:57:21.610212 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:21.610177 2569 generic.go:356] "Generic (PLEG): container finished" podID="b5fa1eac-5061-4a8a-ae23-648aed9d3760" containerID="63a4c676490ecc10f3d645a67c0a607a5ed64ff3f7d6c81baa2839c390295f7e" exitCode=0 Jul 09 18:57:21.610602 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:21.610220 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" event={"ID":"b5fa1eac-5061-4a8a-ae23-648aed9d3760","Type":"ContainerDied","Data":"63a4c676490ecc10f3d645a67c0a607a5ed64ff3f7d6c81baa2839c390295f7e"} Jul 09 18:57:28.644285 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:28.644246 2569 generic.go:356] "Generic (PLEG): container finished" podID="b5fa1eac-5061-4a8a-ae23-648aed9d3760" containerID="902fe599c0ce84279e3fa4c9a4ee37946bad8a780977eb342c115bbf696c41f8" exitCode=0 Jul 09 18:57:28.644617 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:28.644316 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" event={"ID":"b5fa1eac-5061-4a8a-ae23-648aed9d3760","Type":"ContainerDied","Data":"902fe599c0ce84279e3fa4c9a4ee37946bad8a780977eb342c115bbf696c41f8"} Jul 09 18:57:29.780719 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:29.780689 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" Jul 09 18:57:29.899687 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:29.899591 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5fa1eac-5061-4a8a-ae23-648aed9d3760-util\") pod \"b5fa1eac-5061-4a8a-ae23-648aed9d3760\" (UID: \"b5fa1eac-5061-4a8a-ae23-648aed9d3760\") " Jul 09 18:57:29.899687 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:29.899676 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5fa1eac-5061-4a8a-ae23-648aed9d3760-bundle\") pod \"b5fa1eac-5061-4a8a-ae23-648aed9d3760\" (UID: \"b5fa1eac-5061-4a8a-ae23-648aed9d3760\") " Jul 09 18:57:29.899920 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:29.899699 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh4r7\" (UniqueName: \"kubernetes.io/projected/b5fa1eac-5061-4a8a-ae23-648aed9d3760-kube-api-access-zh4r7\") pod \"b5fa1eac-5061-4a8a-ae23-648aed9d3760\" (UID: \"b5fa1eac-5061-4a8a-ae23-648aed9d3760\") " Jul 09 18:57:29.900358 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:29.900332 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5fa1eac-5061-4a8a-ae23-648aed9d3760-bundle" (OuterVolumeSpecName: "bundle") pod "b5fa1eac-5061-4a8a-ae23-648aed9d3760" (UID: "b5fa1eac-5061-4a8a-ae23-648aed9d3760"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:57:29.901855 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:29.901829 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fa1eac-5061-4a8a-ae23-648aed9d3760-kube-api-access-zh4r7" (OuterVolumeSpecName: "kube-api-access-zh4r7") pod "b5fa1eac-5061-4a8a-ae23-648aed9d3760" (UID: "b5fa1eac-5061-4a8a-ae23-648aed9d3760"). InnerVolumeSpecName "kube-api-access-zh4r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:57:29.904534 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:29.904515 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5fa1eac-5061-4a8a-ae23-648aed9d3760-util" (OuterVolumeSpecName: "util") pod "b5fa1eac-5061-4a8a-ae23-648aed9d3760" (UID: "b5fa1eac-5061-4a8a-ae23-648aed9d3760"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:57:30.000562 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:30.000522 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5fa1eac-5061-4a8a-ae23-648aed9d3760-util\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:57:30.000562 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:30.000555 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5fa1eac-5061-4a8a-ae23-648aed9d3760-bundle\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:57:30.000562 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:30.000567 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh4r7\" (UniqueName: \"kubernetes.io/projected/b5fa1eac-5061-4a8a-ae23-648aed9d3760-kube-api-access-zh4r7\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:57:30.654557 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:30.654521 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" event={"ID":"b5fa1eac-5061-4a8a-ae23-648aed9d3760","Type":"ContainerDied","Data":"f7b359232937bc9d3b48db58bedac401ed7b371f406731bd76903d6f16c45b19"} Jul 09 18:57:30.654717 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:30.654573 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7b359232937bc9d3b48db58bedac401ed7b371f406731bd76903d6f16c45b19" Jul 09 18:57:30.654717 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:30.654538 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xm2nl" Jul 09 18:57:35.997763 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:35.997735 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2vqbr"] Jul 09 18:57:35.998150 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:35.998035 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5fa1eac-5061-4a8a-ae23-648aed9d3760" containerName="util" Jul 09 18:57:35.998150 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:35.998047 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5fa1eac-5061-4a8a-ae23-648aed9d3760" containerName="util" Jul 09 18:57:35.998150 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:35.998055 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5fa1eac-5061-4a8a-ae23-648aed9d3760" containerName="extract" Jul 09 18:57:35.998150 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:35.998060 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5fa1eac-5061-4a8a-ae23-648aed9d3760" containerName="extract" Jul 09 18:57:35.998150 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:35.998072 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5fa1eac-5061-4a8a-ae23-648aed9d3760" containerName="pull" Jul 09 18:57:35.998150 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:35.998078 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5fa1eac-5061-4a8a-ae23-648aed9d3760" containerName="pull" Jul 09 18:57:35.998150 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:35.998119 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5fa1eac-5061-4a8a-ae23-648aed9d3760" containerName="extract" Jul 09 18:57:36.034733 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:36.034707 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2vqbr"] Jul 09 18:57:36.034859 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:36.034792 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2vqbr" Jul 09 18:57:36.037438 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:36.037418 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8rcx7\"" Jul 09 18:57:36.037728 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:36.037713 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 09 18:57:36.037971 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:36.037941 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 09 18:57:36.147200 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:36.147168 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15968706-950d-4e0d-ae0f-b5083aeaf81e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2vqbr\" (UID: \"15968706-950d-4e0d-ae0f-b5083aeaf81e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2vqbr" Jul 09 18:57:36.147356 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:36.147208 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldfwr\" (UniqueName: \"kubernetes.io/projected/15968706-950d-4e0d-ae0f-b5083aeaf81e-kube-api-access-ldfwr\") pod \"cert-manager-operator-controller-manager-55c444bf86-2vqbr\" (UID: \"15968706-950d-4e0d-ae0f-b5083aeaf81e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2vqbr" Jul 09 18:57:36.248516 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:36.248436 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldfwr\" (UniqueName: \"kubernetes.io/projected/15968706-950d-4e0d-ae0f-b5083aeaf81e-kube-api-access-ldfwr\") pod \"cert-manager-operator-controller-manager-55c444bf86-2vqbr\" (UID: \"15968706-950d-4e0d-ae0f-b5083aeaf81e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2vqbr" Jul 09 18:57:36.248516 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:36.248513 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15968706-950d-4e0d-ae0f-b5083aeaf81e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2vqbr\" (UID: \"15968706-950d-4e0d-ae0f-b5083aeaf81e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2vqbr" Jul 09 18:57:36.248863 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:36.248847 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15968706-950d-4e0d-ae0f-b5083aeaf81e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2vqbr\" (UID: \"15968706-950d-4e0d-ae0f-b5083aeaf81e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2vqbr" Jul 09 18:57:36.260191 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:36.260161 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldfwr\" (UniqueName: \"kubernetes.io/projected/15968706-950d-4e0d-ae0f-b5083aeaf81e-kube-api-access-ldfwr\") pod \"cert-manager-operator-controller-manager-55c444bf86-2vqbr\" (UID: \"15968706-950d-4e0d-ae0f-b5083aeaf81e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2vqbr" Jul 09 18:57:36.343628 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:36.343599 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2vqbr" Jul 09 18:57:36.499981 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:36.499938 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2vqbr"] Jul 09 18:57:36.502203 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:57:36.502171 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15968706_950d_4e0d_ae0f_b5083aeaf81e.slice/crio-106bb59f116cb3300ea129a085be83e6df7dd88de9e27ebb4d02b0ee916c6336 WatchSource:0}: Error finding container 106bb59f116cb3300ea129a085be83e6df7dd88de9e27ebb4d02b0ee916c6336: Status 404 returned error can't find the container with id 106bb59f116cb3300ea129a085be83e6df7dd88de9e27ebb4d02b0ee916c6336 Jul 09 18:57:36.679922 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:36.679885 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2vqbr" event={"ID":"15968706-950d-4e0d-ae0f-b5083aeaf81e","Type":"ContainerStarted","Data":"106bb59f116cb3300ea129a085be83e6df7dd88de9e27ebb4d02b0ee916c6336"} Jul 09 18:57:39.674051 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:39.674016 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lgtww"] Jul 09 18:57:39.692846 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:39.692819 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lgtww"] Jul 09 18:57:39.693044 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:39.692863 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2vqbr" event={"ID":"15968706-950d-4e0d-ae0f-b5083aeaf81e","Type":"ContainerStarted","Data":"569724692b8098490334e2d4cdcf8960dc3d572f27ea5758f1f8feddf5ddcab7"} Jul 09 18:57:39.693126 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:39.693109 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lgtww" Jul 09 18:57:39.695677 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:39.695658 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-m58hg\"" Jul 09 18:57:39.695794 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:39.695699 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 09 18:57:39.696566 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:39.696551 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 09 18:57:39.715078 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:39.715037 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2vqbr" podStartSLOduration=2.490900123 podStartE2EDuration="4.715019325s" totalImagesPullingTime="2.224119202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:57:39.714495524 +0000 UTC m=+244.539658555" watchObservedRunningTime="2026-07-09 18:57:39.715019325 +0000 UTC m=+244.540182363" Jul 09 18:57:39.778222 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:39.778190 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p68rr\" (UniqueName: \"kubernetes.io/projected/14c4e0d0-1b88-458c-a435-91c640a7911c-kube-api-access-p68rr\") pod \"cert-manager-webhook-6486ffd997-lgtww\" (UID: \"14c4e0d0-1b88-458c-a435-91c640a7911c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lgtww" Jul 09 18:57:39.778400 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:39.778287 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/14c4e0d0-1b88-458c-a435-91c640a7911c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lgtww\" (UID: \"14c4e0d0-1b88-458c-a435-91c640a7911c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lgtww" Jul 09 18:57:39.878773 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:39.878739 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/14c4e0d0-1b88-458c-a435-91c640a7911c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lgtww\" (UID: \"14c4e0d0-1b88-458c-a435-91c640a7911c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lgtww" Jul 09 18:57:39.878967 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:39.878791 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p68rr\" (UniqueName: \"kubernetes.io/projected/14c4e0d0-1b88-458c-a435-91c640a7911c-kube-api-access-p68rr\") pod \"cert-manager-webhook-6486ffd997-lgtww\" (UID: \"14c4e0d0-1b88-458c-a435-91c640a7911c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lgtww" Jul 09 18:57:39.888979 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:39.888920 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/14c4e0d0-1b88-458c-a435-91c640a7911c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lgtww\" (UID: \"14c4e0d0-1b88-458c-a435-91c640a7911c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lgtww" Jul 09 18:57:39.889183 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:39.889167 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p68rr\" (UniqueName: \"kubernetes.io/projected/14c4e0d0-1b88-458c-a435-91c640a7911c-kube-api-access-p68rr\") pod \"cert-manager-webhook-6486ffd997-lgtww\" (UID: \"14c4e0d0-1b88-458c-a435-91c640a7911c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lgtww" Jul 09 18:57:40.015878 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:40.015842 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lgtww" Jul 09 18:57:40.180274 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:40.180251 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lgtww"] Jul 09 18:57:40.181869 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:57:40.181845 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14c4e0d0_1b88_458c_a435_91c640a7911c.slice/crio-96df534aae4571f14b0aa0910d39763e7142a7274e0a4e8138b0add1f0a8bada WatchSource:0}: Error finding container 96df534aae4571f14b0aa0910d39763e7142a7274e0a4e8138b0add1f0a8bada: Status 404 returned error can't find the container with id 96df534aae4571f14b0aa0910d39763e7142a7274e0a4e8138b0add1f0a8bada Jul 09 18:57:40.697641 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:40.697607 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lgtww" event={"ID":"14c4e0d0-1b88-458c-a435-91c640a7911c","Type":"ContainerStarted","Data":"96df534aae4571f14b0aa0910d39763e7142a7274e0a4e8138b0add1f0a8bada"} Jul 09 18:57:40.902554 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:40.902521 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7"] Jul 09 18:57:40.937699 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:40.937673 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7"] Jul 09 18:57:40.937859 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:40.937778 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" Jul 09 18:57:40.940448 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:40.940424 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 18:57:40.940612 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:40.940530 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cd2fs\"" Jul 09 18:57:40.941367 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:40.941352 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 18:57:41.087403 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:41.087364 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnjc5\" (UniqueName: \"kubernetes.io/projected/a95d794e-04b6-411a-b008-dcf808ac33b7-kube-api-access-vnjc5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7\" (UID: \"a95d794e-04b6-411a-b008-dcf808ac33b7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" Jul 09 18:57:41.087403 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:41.087409 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a95d794e-04b6-411a-b008-dcf808ac33b7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7\" (UID: \"a95d794e-04b6-411a-b008-dcf808ac33b7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" Jul 09 18:57:41.087607 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:41.087433 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a95d794e-04b6-411a-b008-dcf808ac33b7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7\" (UID: \"a95d794e-04b6-411a-b008-dcf808ac33b7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" Jul 09 18:57:41.188463 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:41.188405 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnjc5\" (UniqueName: \"kubernetes.io/projected/a95d794e-04b6-411a-b008-dcf808ac33b7-kube-api-access-vnjc5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7\" (UID: \"a95d794e-04b6-411a-b008-dcf808ac33b7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" Jul 09 18:57:41.188463 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:41.188465 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a95d794e-04b6-411a-b008-dcf808ac33b7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7\" (UID: \"a95d794e-04b6-411a-b008-dcf808ac33b7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" Jul 09 18:57:41.188709 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:41.188499 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a95d794e-04b6-411a-b008-dcf808ac33b7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7\" (UID: \"a95d794e-04b6-411a-b008-dcf808ac33b7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" Jul 09 18:57:41.188923 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:41.188899 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a95d794e-04b6-411a-b008-dcf808ac33b7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7\" (UID: \"a95d794e-04b6-411a-b008-dcf808ac33b7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" Jul 09 18:57:41.189044 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:41.189003 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a95d794e-04b6-411a-b008-dcf808ac33b7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7\" (UID: \"a95d794e-04b6-411a-b008-dcf808ac33b7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" Jul 09 18:57:41.202326 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:41.202299 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnjc5\" (UniqueName: \"kubernetes.io/projected/a95d794e-04b6-411a-b008-dcf808ac33b7-kube-api-access-vnjc5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7\" (UID: \"a95d794e-04b6-411a-b008-dcf808ac33b7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" Jul 09 18:57:41.247227 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:41.247193 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" Jul 09 18:57:41.427788 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:41.427711 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7"] Jul 09 18:57:41.430884 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:57:41.430859 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95d794e_04b6_411a_b008_dcf808ac33b7.slice/crio-3121b96dcf0ef21635576475fe91f70c94500d3619831f6f75ba0eb1b7cdbcbc WatchSource:0}: Error finding container 3121b96dcf0ef21635576475fe91f70c94500d3619831f6f75ba0eb1b7cdbcbc: Status 404 returned error can't find the container with id 3121b96dcf0ef21635576475fe91f70c94500d3619831f6f75ba0eb1b7cdbcbc Jul 09 18:57:41.703317 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:41.703275 2569 generic.go:356] "Generic (PLEG): container finished" podID="a95d794e-04b6-411a-b008-dcf808ac33b7" containerID="431fdf27833014fad03cd5cfa17e08eb16e2edef00b6d73258a8688ab5576a48" exitCode=0 Jul 09 18:57:41.703750 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:41.703323 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" event={"ID":"a95d794e-04b6-411a-b008-dcf808ac33b7","Type":"ContainerDied","Data":"431fdf27833014fad03cd5cfa17e08eb16e2edef00b6d73258a8688ab5576a48"} Jul 09 18:57:41.703750 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:41.703370 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" event={"ID":"a95d794e-04b6-411a-b008-dcf808ac33b7","Type":"ContainerStarted","Data":"3121b96dcf0ef21635576475fe91f70c94500d3619831f6f75ba0eb1b7cdbcbc"} Jul 09 18:57:42.076476 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:42.076428 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-68vjf"] Jul 09 18:57:42.078689 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:42.078664 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-68vjf" Jul 09 18:57:42.082007 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:42.081984 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-pgg9w\"" Jul 09 18:57:42.095309 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:42.095282 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-68vjf"] Jul 09 18:57:42.199843 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:42.199806 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b714f56-2944-4db5-9436-a7904f735b4e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-68vjf\" (UID: \"3b714f56-2944-4db5-9436-a7904f735b4e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-68vjf" Jul 09 18:57:42.200007 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:42.199852 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngdd8\" (UniqueName: \"kubernetes.io/projected/3b714f56-2944-4db5-9436-a7904f735b4e-kube-api-access-ngdd8\") pod \"cert-manager-cainjector-559cdd8c87-68vjf\" (UID: \"3b714f56-2944-4db5-9436-a7904f735b4e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-68vjf" Jul 09 18:57:42.301248 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:42.301207 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b714f56-2944-4db5-9436-a7904f735b4e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-68vjf\" (UID: \"3b714f56-2944-4db5-9436-a7904f735b4e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-68vjf" Jul 09 18:57:42.301416 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:42.301254 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngdd8\" (UniqueName: \"kubernetes.io/projected/3b714f56-2944-4db5-9436-a7904f735b4e-kube-api-access-ngdd8\") pod \"cert-manager-cainjector-559cdd8c87-68vjf\" (UID: \"3b714f56-2944-4db5-9436-a7904f735b4e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-68vjf" Jul 09 18:57:42.310318 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:42.310290 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b714f56-2944-4db5-9436-a7904f735b4e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-68vjf\" (UID: \"3b714f56-2944-4db5-9436-a7904f735b4e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-68vjf" Jul 09 18:57:42.310515 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:42.310490 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngdd8\" (UniqueName: \"kubernetes.io/projected/3b714f56-2944-4db5-9436-a7904f735b4e-kube-api-access-ngdd8\") pod \"cert-manager-cainjector-559cdd8c87-68vjf\" (UID: \"3b714f56-2944-4db5-9436-a7904f735b4e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-68vjf" Jul 09 18:57:42.389008 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:42.388907 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-68vjf" Jul 09 18:57:43.254561 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:43.254524 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-68vjf"] Jul 09 18:57:43.256627 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:57:43.256600 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b714f56_2944_4db5_9436_a7904f735b4e.slice/crio-ed9b9647b7a3bec5651cfaee0b7cc360cf38362400fe671c3fc933a00b7919bf WatchSource:0}: Error finding container ed9b9647b7a3bec5651cfaee0b7cc360cf38362400fe671c3fc933a00b7919bf: Status 404 returned error can't find the container with id ed9b9647b7a3bec5651cfaee0b7cc360cf38362400fe671c3fc933a00b7919bf Jul 09 18:57:43.714929 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:43.714855 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lgtww" event={"ID":"14c4e0d0-1b88-458c-a435-91c640a7911c","Type":"ContainerStarted","Data":"54cea2415ba69ff2cc4a7d5af341d82eb40c9c333e09f81cd9f844fc1beafa70"} Jul 09 18:57:43.714929 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:43.714910 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lgtww" Jul 09 18:57:43.716643 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:43.716620 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-68vjf" event={"ID":"3b714f56-2944-4db5-9436-a7904f735b4e","Type":"ContainerStarted","Data":"a45d95dabcef21aa8f77f5f798b90738c74b44ea1960a03af02ad9478e578c31"} Jul 09 18:57:43.716758 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:43.716650 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-68vjf" event={"ID":"3b714f56-2944-4db5-9436-a7904f735b4e","Type":"ContainerStarted","Data":"ed9b9647b7a3bec5651cfaee0b7cc360cf38362400fe671c3fc933a00b7919bf"} Jul 09 18:57:43.735605 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:43.735558 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-lgtww" podStartSLOduration=1.739897338 podStartE2EDuration="4.735542037s" totalImagesPullingTime="2.995644699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:57:43.734994909 +0000 UTC m=+248.560157945" watchObservedRunningTime="2026-07-09 18:57:43.735542037 +0000 UTC m=+248.560705073" Jul 09 18:57:43.754812 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:43.754758 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-68vjf" podStartSLOduration=1.754742011 podStartE2EDuration="1.754742011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:57:43.752824877 +0000 UTC m=+248.577987927" watchObservedRunningTime="2026-07-09 18:57:43.754742011 +0000 UTC m=+248.579905047" Jul 09 18:57:45.726896 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:45.726859 2569 generic.go:356] "Generic (PLEG): container finished" podID="a95d794e-04b6-411a-b008-dcf808ac33b7" containerID="9776ebe67f75c83c48af017281a4122f41f8cd713dc06b4f716d3f48f29cb726" exitCode=0 Jul 09 18:57:45.727339 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:45.726969 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" event={"ID":"a95d794e-04b6-411a-b008-dcf808ac33b7","Type":"ContainerDied","Data":"9776ebe67f75c83c48af017281a4122f41f8cd713dc06b4f716d3f48f29cb726"} Jul 09 18:57:46.732526 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:46.732488 2569 generic.go:356] "Generic (PLEG): container finished" podID="a95d794e-04b6-411a-b008-dcf808ac33b7" containerID="d55d806dd3b92a0c9166d24bd484ae56b92f26b3802c62b508f08ca7cb909c70" exitCode=0 Jul 09 18:57:46.732871 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:46.732574 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" event={"ID":"a95d794e-04b6-411a-b008-dcf808ac33b7","Type":"ContainerDied","Data":"d55d806dd3b92a0c9166d24bd484ae56b92f26b3802c62b508f08ca7cb909c70"} Jul 09 18:57:47.860119 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:47.860094 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" Jul 09 18:57:47.950746 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:47.950710 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a95d794e-04b6-411a-b008-dcf808ac33b7-bundle\") pod \"a95d794e-04b6-411a-b008-dcf808ac33b7\" (UID: \"a95d794e-04b6-411a-b008-dcf808ac33b7\") " Jul 09 18:57:47.950879 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:47.950812 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnjc5\" (UniqueName: \"kubernetes.io/projected/a95d794e-04b6-411a-b008-dcf808ac33b7-kube-api-access-vnjc5\") pod \"a95d794e-04b6-411a-b008-dcf808ac33b7\" (UID: \"a95d794e-04b6-411a-b008-dcf808ac33b7\") " Jul 09 18:57:47.950879 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:47.950864 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a95d794e-04b6-411a-b008-dcf808ac33b7-util\") pod \"a95d794e-04b6-411a-b008-dcf808ac33b7\" (UID: \"a95d794e-04b6-411a-b008-dcf808ac33b7\") " Jul 09 18:57:47.951195 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:47.951172 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95d794e-04b6-411a-b008-dcf808ac33b7-bundle" (OuterVolumeSpecName: "bundle") pod "a95d794e-04b6-411a-b008-dcf808ac33b7" (UID: "a95d794e-04b6-411a-b008-dcf808ac33b7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:57:47.952897 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:47.952867 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95d794e-04b6-411a-b008-dcf808ac33b7-kube-api-access-vnjc5" (OuterVolumeSpecName: "kube-api-access-vnjc5") pod "a95d794e-04b6-411a-b008-dcf808ac33b7" (UID: "a95d794e-04b6-411a-b008-dcf808ac33b7"). InnerVolumeSpecName "kube-api-access-vnjc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:57:47.955236 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:47.955211 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95d794e-04b6-411a-b008-dcf808ac33b7-util" (OuterVolumeSpecName: "util") pod "a95d794e-04b6-411a-b008-dcf808ac33b7" (UID: "a95d794e-04b6-411a-b008-dcf808ac33b7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:57:48.051763 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:48.051689 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnjc5\" (UniqueName: \"kubernetes.io/projected/a95d794e-04b6-411a-b008-dcf808ac33b7-kube-api-access-vnjc5\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:57:48.051763 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:48.051716 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a95d794e-04b6-411a-b008-dcf808ac33b7-util\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:57:48.051763 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:48.051726 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a95d794e-04b6-411a-b008-dcf808ac33b7-bundle\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:57:48.742160 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:48.742128 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" Jul 09 18:57:48.742160 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:48.742145 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwb7m7" event={"ID":"a95d794e-04b6-411a-b008-dcf808ac33b7","Type":"ContainerDied","Data":"3121b96dcf0ef21635576475fe91f70c94500d3619831f6f75ba0eb1b7cdbcbc"} Jul 09 18:57:48.742428 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:48.742184 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3121b96dcf0ef21635576475fe91f70c94500d3619831f6f75ba0eb1b7cdbcbc" Jul 09 18:57:49.724330 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:57:49.724305 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lgtww" Jul 09 18:58:00.620076 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.619993 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv"] Jul 09 18:58:00.620566 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.620277 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a95d794e-04b6-411a-b008-dcf808ac33b7" containerName="util" Jul 09 18:58:00.620566 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.620287 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a95d794e-04b6-411a-b008-dcf808ac33b7" containerName="util" Jul 09 18:58:00.620566 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.620300 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a95d794e-04b6-411a-b008-dcf808ac33b7" containerName="extract" Jul 09 18:58:00.620566 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.620306 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a95d794e-04b6-411a-b008-dcf808ac33b7" containerName="extract" Jul 09 18:58:00.620566 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.620314 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a95d794e-04b6-411a-b008-dcf808ac33b7" containerName="pull" Jul 09 18:58:00.620566 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.620320 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a95d794e-04b6-411a-b008-dcf808ac33b7" containerName="pull" Jul 09 18:58:00.620566 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.620379 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a95d794e-04b6-411a-b008-dcf808ac33b7" containerName="extract" Jul 09 18:58:00.622764 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.622746 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" Jul 09 18:58:00.625456 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.625435 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 18:58:00.625579 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.625493 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cd2fs\"" Jul 09 18:58:00.626276 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.626256 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 18:58:00.632613 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.632591 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv"] Jul 09 18:58:00.647320 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.647294 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/620181fa-8a30-4edd-a814-590762712172-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv\" (UID: \"620181fa-8a30-4edd-a814-590762712172\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" Jul 09 18:58:00.647405 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.647332 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfhzl\" (UniqueName: \"kubernetes.io/projected/620181fa-8a30-4edd-a814-590762712172-kube-api-access-tfhzl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv\" (UID: \"620181fa-8a30-4edd-a814-590762712172\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" Jul 09 18:58:00.647405 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.647354 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/620181fa-8a30-4edd-a814-590762712172-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv\" (UID: \"620181fa-8a30-4edd-a814-590762712172\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" Jul 09 18:58:00.748206 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.748170 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/620181fa-8a30-4edd-a814-590762712172-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv\" (UID: \"620181fa-8a30-4edd-a814-590762712172\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" Jul 09 18:58:00.748375 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.748217 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfhzl\" (UniqueName: \"kubernetes.io/projected/620181fa-8a30-4edd-a814-590762712172-kube-api-access-tfhzl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv\" (UID: \"620181fa-8a30-4edd-a814-590762712172\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" Jul 09 18:58:00.748375 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.748342 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/620181fa-8a30-4edd-a814-590762712172-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv\" (UID: \"620181fa-8a30-4edd-a814-590762712172\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" Jul 09 18:58:00.748641 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.748621 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/620181fa-8a30-4edd-a814-590762712172-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv\" (UID: \"620181fa-8a30-4edd-a814-590762712172\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" Jul 09 18:58:00.748682 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.748657 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/620181fa-8a30-4edd-a814-590762712172-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv\" (UID: \"620181fa-8a30-4edd-a814-590762712172\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" Jul 09 18:58:00.758740 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.758710 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfhzl\" (UniqueName: \"kubernetes.io/projected/620181fa-8a30-4edd-a814-590762712172-kube-api-access-tfhzl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv\" (UID: \"620181fa-8a30-4edd-a814-590762712172\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" Jul 09 18:58:00.932215 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:00.932133 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" Jul 09 18:58:01.058359 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:01.058333 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv"] Jul 09 18:58:01.060775 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:58:01.060743 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620181fa_8a30_4edd_a814_590762712172.slice/crio-7ea40b2787c7afbcce4e4a2585076dd464389bcc05522a2a0b12514670aaae2e WatchSource:0}: Error finding container 7ea40b2787c7afbcce4e4a2585076dd464389bcc05522a2a0b12514670aaae2e: Status 404 returned error can't find the container with id 7ea40b2787c7afbcce4e4a2585076dd464389bcc05522a2a0b12514670aaae2e Jul 09 18:58:01.798362 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:01.798330 2569 generic.go:356] "Generic (PLEG): container finished" podID="620181fa-8a30-4edd-a814-590762712172" containerID="faf3a85f66c33aa03ab1bee7b2737f0212f90cb035b404921976b2fc21081c76" exitCode=0 Jul 09 18:58:01.798741 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:01.798417 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" event={"ID":"620181fa-8a30-4edd-a814-590762712172","Type":"ContainerDied","Data":"faf3a85f66c33aa03ab1bee7b2737f0212f90cb035b404921976b2fc21081c76"} Jul 09 18:58:01.798741 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:01.798493 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" event={"ID":"620181fa-8a30-4edd-a814-590762712172","Type":"ContainerStarted","Data":"7ea40b2787c7afbcce4e4a2585076dd464389bcc05522a2a0b12514670aaae2e"} Jul 09 18:58:02.803740 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:02.803705 2569 generic.go:356] "Generic (PLEG): container finished" podID="620181fa-8a30-4edd-a814-590762712172" containerID="275a53ff4a2f69a46e83b0256265f5fbf362868dd22430b201735ce5250e836c" exitCode=0 Jul 09 18:58:02.804120 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:02.803759 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" event={"ID":"620181fa-8a30-4edd-a814-590762712172","Type":"ContainerDied","Data":"275a53ff4a2f69a46e83b0256265f5fbf362868dd22430b201735ce5250e836c"} Jul 09 18:58:03.809490 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:03.809452 2569 generic.go:356] "Generic (PLEG): container finished" podID="620181fa-8a30-4edd-a814-590762712172" containerID="26b089c351d0aa3131a0cce445d580a35b0b50420a9de5f79cd1752336a3f3f4" exitCode=0 Jul 09 18:58:03.809867 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:03.809501 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" event={"ID":"620181fa-8a30-4edd-a814-590762712172","Type":"ContainerDied","Data":"26b089c351d0aa3131a0cce445d580a35b0b50420a9de5f79cd1752336a3f3f4"} Jul 09 18:58:04.947784 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:04.947761 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" Jul 09 18:58:04.987252 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:04.987229 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/620181fa-8a30-4edd-a814-590762712172-bundle\") pod \"620181fa-8a30-4edd-a814-590762712172\" (UID: \"620181fa-8a30-4edd-a814-590762712172\") " Jul 09 18:58:04.987407 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:04.987264 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/620181fa-8a30-4edd-a814-590762712172-util\") pod \"620181fa-8a30-4edd-a814-590762712172\" (UID: \"620181fa-8a30-4edd-a814-590762712172\") " Jul 09 18:58:04.987407 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:04.987293 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfhzl\" (UniqueName: \"kubernetes.io/projected/620181fa-8a30-4edd-a814-590762712172-kube-api-access-tfhzl\") pod \"620181fa-8a30-4edd-a814-590762712172\" (UID: \"620181fa-8a30-4edd-a814-590762712172\") " Jul 09 18:58:04.987941 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:04.987919 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620181fa-8a30-4edd-a814-590762712172-bundle" (OuterVolumeSpecName: "bundle") pod "620181fa-8a30-4edd-a814-590762712172" (UID: "620181fa-8a30-4edd-a814-590762712172"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:58:04.989275 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:04.989253 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620181fa-8a30-4edd-a814-590762712172-kube-api-access-tfhzl" (OuterVolumeSpecName: "kube-api-access-tfhzl") pod "620181fa-8a30-4edd-a814-590762712172" (UID: "620181fa-8a30-4edd-a814-590762712172"). InnerVolumeSpecName "kube-api-access-tfhzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:58:04.993135 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:04.993020 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620181fa-8a30-4edd-a814-590762712172-util" (OuterVolumeSpecName: "util") pod "620181fa-8a30-4edd-a814-590762712172" (UID: "620181fa-8a30-4edd-a814-590762712172"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:58:05.088347 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:05.088253 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/620181fa-8a30-4edd-a814-590762712172-bundle\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:58:05.088347 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:05.088293 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/620181fa-8a30-4edd-a814-590762712172-util\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:58:05.088347 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:05.088307 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfhzl\" (UniqueName: \"kubernetes.io/projected/620181fa-8a30-4edd-a814-590762712172-kube-api-access-tfhzl\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:58:05.819794 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:05.819764 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" Jul 09 18:58:05.819794 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:05.819788 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4hgvv" event={"ID":"620181fa-8a30-4edd-a814-590762712172","Type":"ContainerDied","Data":"7ea40b2787c7afbcce4e4a2585076dd464389bcc05522a2a0b12514670aaae2e"} Jul 09 18:58:05.820047 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:05.819834 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ea40b2787c7afbcce4e4a2585076dd464389bcc05522a2a0b12514670aaae2e" Jul 09 18:58:13.501161 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.501125 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8"] Jul 09 18:58:13.501548 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.501392 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="620181fa-8a30-4edd-a814-590762712172" containerName="extract" Jul 09 18:58:13.501548 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.501402 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="620181fa-8a30-4edd-a814-590762712172" containerName="extract" Jul 09 18:58:13.501548 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.501423 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="620181fa-8a30-4edd-a814-590762712172" containerName="pull" Jul 09 18:58:13.501548 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.501428 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="620181fa-8a30-4edd-a814-590762712172" containerName="pull" Jul 09 18:58:13.501548 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.501436 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="620181fa-8a30-4edd-a814-590762712172" containerName="util" Jul 09 18:58:13.501548 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.501441 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="620181fa-8a30-4edd-a814-590762712172" containerName="util" Jul 09 18:58:13.501548 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.501483 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="620181fa-8a30-4edd-a814-590762712172" containerName="extract" Jul 09 18:58:13.529174 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.529140 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8"] Jul 09 18:58:13.529337 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.529279 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.534343 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.534308 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 18:58:13.534512 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.534440 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 18:58:13.534825 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.534805 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xxtqd\"" Jul 09 18:58:13.535080 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.534807 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 18:58:13.535184 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.535132 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 18:58:13.537758 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.537736 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11642f9a-aa84-47c8-b35d-1f4aa0397211-tmp\") pod \"opendatahub-operator-controller-manager-5cd98cb899-p9bm8\" (UID: \"11642f9a-aa84-47c8-b35d-1f4aa0397211\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.537876 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.537769 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11642f9a-aa84-47c8-b35d-1f4aa0397211-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-p9bm8\" (UID: \"11642f9a-aa84-47c8-b35d-1f4aa0397211\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.537876 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.537789 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kcbn\" (UniqueName: \"kubernetes.io/projected/11642f9a-aa84-47c8-b35d-1f4aa0397211-kube-api-access-7kcbn\") pod \"opendatahub-operator-controller-manager-5cd98cb899-p9bm8\" (UID: \"11642f9a-aa84-47c8-b35d-1f4aa0397211\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.537876 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.537813 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11642f9a-aa84-47c8-b35d-1f4aa0397211-manifests\") pod \"opendatahub-operator-controller-manager-5cd98cb899-p9bm8\" (UID: \"11642f9a-aa84-47c8-b35d-1f4aa0397211\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.537876 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.537834 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11642f9a-aa84-47c8-b35d-1f4aa0397211-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-p9bm8\" (UID: \"11642f9a-aa84-47c8-b35d-1f4aa0397211\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.638155 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.638123 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11642f9a-aa84-47c8-b35d-1f4aa0397211-tmp\") pod \"opendatahub-operator-controller-manager-5cd98cb899-p9bm8\" (UID: \"11642f9a-aa84-47c8-b35d-1f4aa0397211\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.638301 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.638165 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11642f9a-aa84-47c8-b35d-1f4aa0397211-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-p9bm8\" (UID: \"11642f9a-aa84-47c8-b35d-1f4aa0397211\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.638301 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.638187 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kcbn\" (UniqueName: \"kubernetes.io/projected/11642f9a-aa84-47c8-b35d-1f4aa0397211-kube-api-access-7kcbn\") pod \"opendatahub-operator-controller-manager-5cd98cb899-p9bm8\" (UID: \"11642f9a-aa84-47c8-b35d-1f4aa0397211\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.638301 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.638210 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11642f9a-aa84-47c8-b35d-1f4aa0397211-manifests\") pod \"opendatahub-operator-controller-manager-5cd98cb899-p9bm8\" (UID: \"11642f9a-aa84-47c8-b35d-1f4aa0397211\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.638301 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.638231 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11642f9a-aa84-47c8-b35d-1f4aa0397211-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-p9bm8\" (UID: \"11642f9a-aa84-47c8-b35d-1f4aa0397211\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.638877 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.638842 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11642f9a-aa84-47c8-b35d-1f4aa0397211-manifests\") pod \"opendatahub-operator-controller-manager-5cd98cb899-p9bm8\" (UID: \"11642f9a-aa84-47c8-b35d-1f4aa0397211\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.643997 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.641289 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11642f9a-aa84-47c8-b35d-1f4aa0397211-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-p9bm8\" (UID: \"11642f9a-aa84-47c8-b35d-1f4aa0397211\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.643997 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.643132 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11642f9a-aa84-47c8-b35d-1f4aa0397211-tmp\") pod \"opendatahub-operator-controller-manager-5cd98cb899-p9bm8\" (UID: \"11642f9a-aa84-47c8-b35d-1f4aa0397211\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.644381 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.644353 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11642f9a-aa84-47c8-b35d-1f4aa0397211-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-p9bm8\" (UID: \"11642f9a-aa84-47c8-b35d-1f4aa0397211\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.652154 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.652129 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kcbn\" (UniqueName: \"kubernetes.io/projected/11642f9a-aa84-47c8-b35d-1f4aa0397211-kube-api-access-7kcbn\") pod \"opendatahub-operator-controller-manager-5cd98cb899-p9bm8\" (UID: \"11642f9a-aa84-47c8-b35d-1f4aa0397211\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:13.839692 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:13.839609 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:14.018075 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:14.018045 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8"] Jul 09 18:58:14.021722 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:58:14.021676 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11642f9a_aa84_47c8_b35d_1f4aa0397211.slice/crio-66650e5506004ea22eb835b8ad07da7a6c9ee06e67faf5466a0dc0c9b8aad0a5 WatchSource:0}: Error finding container 66650e5506004ea22eb835b8ad07da7a6c9ee06e67faf5466a0dc0c9b8aad0a5: Status 404 returned error can't find the container with id 66650e5506004ea22eb835b8ad07da7a6c9ee06e67faf5466a0dc0c9b8aad0a5 Jul 09 18:58:14.858321 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:14.858277 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" event={"ID":"11642f9a-aa84-47c8-b35d-1f4aa0397211","Type":"ContainerStarted","Data":"66650e5506004ea22eb835b8ad07da7a6c9ee06e67faf5466a0dc0c9b8aad0a5"} Jul 09 18:58:17.873172 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:17.873138 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" event={"ID":"11642f9a-aa84-47c8-b35d-1f4aa0397211","Type":"ContainerStarted","Data":"e6b91dd9becc36e3bacbd93728d5fdd6bc8d3a2ffa51cef417cf8ef7ff1892b3"} Jul 09 18:58:18.878959 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:18.878921 2569 generic.go:356] "Generic (PLEG): container finished" podID="11642f9a-aa84-47c8-b35d-1f4aa0397211" containerID="e6b91dd9becc36e3bacbd93728d5fdd6bc8d3a2ffa51cef417cf8ef7ff1892b3" exitCode=0 Jul 09 18:58:18.879417 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:18.879008 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" event={"ID":"11642f9a-aa84-47c8-b35d-1f4aa0397211","Type":"ContainerDied","Data":"e6b91dd9becc36e3bacbd93728d5fdd6bc8d3a2ffa51cef417cf8ef7ff1892b3"} Jul 09 18:58:19.884725 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:19.884693 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" event={"ID":"11642f9a-aa84-47c8-b35d-1f4aa0397211","Type":"ContainerStarted","Data":"dee1df3d92a264356ca1f4d4d7a01ce3d3818d037ea8bcf90e6d5219499a9dd4"} Jul 09 18:58:19.885113 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:19.884977 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:19.909578 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:19.909528 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" podStartSLOduration=2.613959877 podStartE2EDuration="6.909512472s" totalImagesPullingTime="3.213114447s" totalInitContainerRuntime="1.082438148s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:19.907621323 +0000 UTC m=+284.732784360" watchObservedRunningTime="2026-07-09 18:58:19.909512472 +0000 UTC m=+284.734675564" Jul 09 18:58:30.892567 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:30.892540 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-p9bm8" Jul 09 18:58:31.364209 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.364177 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs"] Jul 09 18:58:31.367452 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.367436 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:31.370013 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.369993 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 18:58:31.370900 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.370881 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-w7f7f\"" Jul 09 18:58:31.371052 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.371032 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 18:58:31.371170 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.371054 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 18:58:31.371170 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.371063 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 18:58:31.371289 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.371275 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 18:58:31.380263 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.380234 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs"] Jul 09 18:58:31.492185 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.492142 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21e2ff0-3336-4eb7-83b1-fc3a2212935f-cert\") pod \"lws-controller-manager-9dfc7f7f4-fnqqs\" (UID: \"d21e2ff0-3336-4eb7-83b1-fc3a2212935f\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:31.492369 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.492199 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjtt9\" (UniqueName: \"kubernetes.io/projected/d21e2ff0-3336-4eb7-83b1-fc3a2212935f-kube-api-access-bjtt9\") pod \"lws-controller-manager-9dfc7f7f4-fnqqs\" (UID: \"d21e2ff0-3336-4eb7-83b1-fc3a2212935f\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:31.492369 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.492222 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d21e2ff0-3336-4eb7-83b1-fc3a2212935f-metrics-cert\") pod \"lws-controller-manager-9dfc7f7f4-fnqqs\" (UID: \"d21e2ff0-3336-4eb7-83b1-fc3a2212935f\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:31.492369 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.492240 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d21e2ff0-3336-4eb7-83b1-fc3a2212935f-manager-config\") pod \"lws-controller-manager-9dfc7f7f4-fnqqs\" (UID: \"d21e2ff0-3336-4eb7-83b1-fc3a2212935f\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:31.593589 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.593538 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d21e2ff0-3336-4eb7-83b1-fc3a2212935f-manager-config\") pod \"lws-controller-manager-9dfc7f7f4-fnqqs\" (UID: \"d21e2ff0-3336-4eb7-83b1-fc3a2212935f\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:31.593770 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.593627 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21e2ff0-3336-4eb7-83b1-fc3a2212935f-cert\") pod \"lws-controller-manager-9dfc7f7f4-fnqqs\" (UID: \"d21e2ff0-3336-4eb7-83b1-fc3a2212935f\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:31.593770 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.593660 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjtt9\" (UniqueName: \"kubernetes.io/projected/d21e2ff0-3336-4eb7-83b1-fc3a2212935f-kube-api-access-bjtt9\") pod \"lws-controller-manager-9dfc7f7f4-fnqqs\" (UID: \"d21e2ff0-3336-4eb7-83b1-fc3a2212935f\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:31.593770 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.593677 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d21e2ff0-3336-4eb7-83b1-fc3a2212935f-metrics-cert\") pod \"lws-controller-manager-9dfc7f7f4-fnqqs\" (UID: \"d21e2ff0-3336-4eb7-83b1-fc3a2212935f\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:31.594330 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.594305 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d21e2ff0-3336-4eb7-83b1-fc3a2212935f-manager-config\") pod \"lws-controller-manager-9dfc7f7f4-fnqqs\" (UID: \"d21e2ff0-3336-4eb7-83b1-fc3a2212935f\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:31.596138 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.596121 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21e2ff0-3336-4eb7-83b1-fc3a2212935f-cert\") pod \"lws-controller-manager-9dfc7f7f4-fnqqs\" (UID: \"d21e2ff0-3336-4eb7-83b1-fc3a2212935f\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:31.596294 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.596274 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d21e2ff0-3336-4eb7-83b1-fc3a2212935f-metrics-cert\") pod \"lws-controller-manager-9dfc7f7f4-fnqqs\" (UID: \"d21e2ff0-3336-4eb7-83b1-fc3a2212935f\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:31.606063 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.606045 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjtt9\" (UniqueName: \"kubernetes.io/projected/d21e2ff0-3336-4eb7-83b1-fc3a2212935f-kube-api-access-bjtt9\") pod \"lws-controller-manager-9dfc7f7f4-fnqqs\" (UID: \"d21e2ff0-3336-4eb7-83b1-fc3a2212935f\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:31.678859 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.678772 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:31.808918 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.808891 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs"] Jul 09 18:58:31.811310 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:58:31.811285 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21e2ff0_3336_4eb7_83b1_fc3a2212935f.slice/crio-2c4a98e769bf57a7701d03e594ebe260d61a23f811cc02c847e8ff763d095430 WatchSource:0}: Error finding container 2c4a98e769bf57a7701d03e594ebe260d61a23f811cc02c847e8ff763d095430: Status 404 returned error can't find the container with id 2c4a98e769bf57a7701d03e594ebe260d61a23f811cc02c847e8ff763d095430 Jul 09 18:58:31.944322 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:31.944224 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" event={"ID":"d21e2ff0-3336-4eb7-83b1-fc3a2212935f","Type":"ContainerStarted","Data":"2c4a98e769bf57a7701d03e594ebe260d61a23f811cc02c847e8ff763d095430"} Jul 09 18:58:33.607891 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.607849 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q"] Jul 09 18:58:33.611642 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.611606 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q" Jul 09 18:58:33.616032 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.616009 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 18:58:33.617338 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.617321 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 18:58:33.619137 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.619114 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 18:58:33.620627 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.620607 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 18:58:33.625206 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.625188 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-56g44\"" Jul 09 18:58:33.655436 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.655411 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q"] Jul 09 18:58:33.711757 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.711717 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73e08385-5634-4158-928d-7c840cc3216b-tls-certs\") pod \"kube-auth-proxy-5b58cdb447-4kr8q\" (UID: \"73e08385-5634-4158-928d-7c840cc3216b\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q" Jul 09 18:58:33.711926 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.711834 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73e08385-5634-4158-928d-7c840cc3216b-tmp\") pod \"kube-auth-proxy-5b58cdb447-4kr8q\" (UID: \"73e08385-5634-4158-928d-7c840cc3216b\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q" Jul 09 18:58:33.711926 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.711867 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqgz8\" (UniqueName: \"kubernetes.io/projected/73e08385-5634-4158-928d-7c840cc3216b-kube-api-access-rqgz8\") pod \"kube-auth-proxy-5b58cdb447-4kr8q\" (UID: \"73e08385-5634-4158-928d-7c840cc3216b\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q" Jul 09 18:58:33.812990 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.812918 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73e08385-5634-4158-928d-7c840cc3216b-tmp\") pod \"kube-auth-proxy-5b58cdb447-4kr8q\" (UID: \"73e08385-5634-4158-928d-7c840cc3216b\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q" Jul 09 18:58:33.812990 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.812993 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqgz8\" (UniqueName: \"kubernetes.io/projected/73e08385-5634-4158-928d-7c840cc3216b-kube-api-access-rqgz8\") pod \"kube-auth-proxy-5b58cdb447-4kr8q\" (UID: \"73e08385-5634-4158-928d-7c840cc3216b\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q" Jul 09 18:58:33.813241 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.813060 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73e08385-5634-4158-928d-7c840cc3216b-tls-certs\") pod \"kube-auth-proxy-5b58cdb447-4kr8q\" (UID: \"73e08385-5634-4158-928d-7c840cc3216b\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q" Jul 09 18:58:33.815719 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.815690 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73e08385-5634-4158-928d-7c840cc3216b-tmp\") pod \"kube-auth-proxy-5b58cdb447-4kr8q\" (UID: \"73e08385-5634-4158-928d-7c840cc3216b\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q" Jul 09 18:58:33.816057 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.816006 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73e08385-5634-4158-928d-7c840cc3216b-tls-certs\") pod \"kube-auth-proxy-5b58cdb447-4kr8q\" (UID: \"73e08385-5634-4158-928d-7c840cc3216b\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q" Jul 09 18:58:33.831511 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.831479 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqgz8\" (UniqueName: \"kubernetes.io/projected/73e08385-5634-4158-928d-7c840cc3216b-kube-api-access-rqgz8\") pod \"kube-auth-proxy-5b58cdb447-4kr8q\" (UID: \"73e08385-5634-4158-928d-7c840cc3216b\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q" Jul 09 18:58:33.923220 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:33.923192 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q" Jul 09 18:58:34.068970 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:34.068894 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q"] Jul 09 18:58:34.070840 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:58:34.070817 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e08385_5634_4158_928d_7c840cc3216b.slice/crio-9a39f4f43e74cfbb56b98e44ba87dc3ff92a3a30c42e26c1c8a75e3148757e83 WatchSource:0}: Error finding container 9a39f4f43e74cfbb56b98e44ba87dc3ff92a3a30c42e26c1c8a75e3148757e83: Status 404 returned error can't find the container with id 9a39f4f43e74cfbb56b98e44ba87dc3ff92a3a30c42e26c1c8a75e3148757e83 Jul 09 18:58:34.964852 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:34.964781 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" event={"ID":"d21e2ff0-3336-4eb7-83b1-fc3a2212935f","Type":"ContainerStarted","Data":"b596b183189abf35d1eaa3056d884515baa2b1880d4dc1022f9602e054af3262"} Jul 09 18:58:34.965366 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:34.965050 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:34.966449 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:34.966421 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q" event={"ID":"73e08385-5634-4158-928d-7c840cc3216b","Type":"ContainerStarted","Data":"9a39f4f43e74cfbb56b98e44ba87dc3ff92a3a30c42e26c1c8a75e3148757e83"} Jul 09 18:58:35.002268 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:35.002205 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" podStartSLOduration=1.9007310689999999 podStartE2EDuration="4.002184862s" totalImagesPullingTime="2.101453793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:35.001518699 +0000 UTC m=+299.826681738" watchObservedRunningTime="2026-07-09 18:58:35.002184862 +0000 UTC m=+299.827347899" Jul 09 18:58:35.572845 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:35.572817 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 18:58:35.578616 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:35.578593 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 18:58:35.579997 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:35.579974 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 18:58:37.104089 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:37.104065 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 18:58:37.982572 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:37.982535 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q" event={"ID":"73e08385-5634-4158-928d-7c840cc3216b","Type":"ContainerStarted","Data":"3d62db11e452cc535225893b2bfbb7cd7cb4cd0fa3721f94f27531f7b3adebef"} Jul 09 18:58:38.018836 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:38.018789 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b58cdb447-4kr8q" podStartSLOduration=1.991711909 podStartE2EDuration="5.01877379s" totalImagesPullingTime="3.027061881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:38.017739036 +0000 UTC m=+302.842902075" watchObservedRunningTime="2026-07-09 18:58:38.01877379 +0000 UTC m=+302.843936826" Jul 09 18:58:44.264974 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:44.264917 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk"] Jul 09 18:58:44.268372 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:44.268356 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk" Jul 09 18:58:44.276052 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:44.276030 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-whnrt\"" Jul 09 18:58:44.285792 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:44.285775 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 09 18:58:44.297817 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:44.297791 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk"] Jul 09 18:58:44.400055 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:44.400014 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-698jk\" (UID: \"20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk" Jul 09 18:58:44.400055 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:44.400055 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2mjs\" (UniqueName: \"kubernetes.io/projected/20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4-kube-api-access-t2mjs\") pod \"llmisvc-controller-manager-777c6bd7d5-698jk\" (UID: \"20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk" Jul 09 18:58:44.501093 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:44.501054 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-698jk\" (UID: \"20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk" Jul 09 18:58:44.501093 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:44.501092 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2mjs\" (UniqueName: \"kubernetes.io/projected/20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4-kube-api-access-t2mjs\") pod \"llmisvc-controller-manager-777c6bd7d5-698jk\" (UID: \"20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk" Jul 09 18:58:44.503462 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:44.503441 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-698jk\" (UID: \"20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk" Jul 09 18:58:44.511273 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:44.511250 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2mjs\" (UniqueName: \"kubernetes.io/projected/20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4-kube-api-access-t2mjs\") pod \"llmisvc-controller-manager-777c6bd7d5-698jk\" (UID: \"20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk" Jul 09 18:58:44.577617 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:44.577537 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk" Jul 09 18:58:44.727693 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:44.727671 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk"] Jul 09 18:58:44.729927 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:58:44.729897 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod20d3f3c6_078c_4b86_bd6d_d916d0f2cdf4.slice/crio-f67d33447c909b5d975d61a99b2771c6c6e2b5171344f6bc5759aa1ed2cc3164 WatchSource:0}: Error finding container f67d33447c909b5d975d61a99b2771c6c6e2b5171344f6bc5759aa1ed2cc3164: Status 404 returned error can't find the container with id f67d33447c909b5d975d61a99b2771c6c6e2b5171344f6bc5759aa1ed2cc3164 Jul 09 18:58:44.731197 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:44.731181 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 18:58:45.015248 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:45.015214 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk" event={"ID":"20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4","Type":"ContainerStarted","Data":"f67d33447c909b5d975d61a99b2771c6c6e2b5171344f6bc5759aa1ed2cc3164"} Jul 09 18:58:45.974454 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:45.974419 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-fnqqs" Jul 09 18:58:48.030514 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:48.030484 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk" event={"ID":"20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4","Type":"ContainerStarted","Data":"c67526b627f3d52b1e87f9a43e7079b4a004cd60eb9e69afdb055d0b5b73a76c"} Jul 09 18:58:48.030889 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:48.030559 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk" Jul 09 18:58:48.069410 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:58:48.069315 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk" podStartSLOduration=0.998193803 podStartE2EDuration="4.069299167s" totalImagesPullingTime="3.071105364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:48.066765949 +0000 UTC m=+312.891928986" watchObservedRunningTime="2026-07-09 18:58:48.069299167 +0000 UTC m=+312.894462204" Jul 09 18:59:19.038085 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:19.038048 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-698jk" Jul 09 18:59:46.819295 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:46.819259 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv"] Jul 09 18:59:46.821840 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:46.821821 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" Jul 09 18:59:46.824445 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:46.824426 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xcftl\"" Jul 09 18:59:46.824543 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:46.824447 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 18:59:46.825351 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:46.825331 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 18:59:46.831816 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:46.831793 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv"] Jul 09 18:59:46.923580 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:46.923547 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aa80757-737d-4805-981a-e102986f8479-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv\" (UID: \"7aa80757-737d-4805-981a-e102986f8479\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" Jul 09 18:59:46.923748 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:46.923603 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxwmz\" (UniqueName: \"kubernetes.io/projected/7aa80757-737d-4805-981a-e102986f8479-kube-api-access-mxwmz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv\" (UID: \"7aa80757-737d-4805-981a-e102986f8479\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" Jul 09 18:59:46.923748 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:46.923695 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aa80757-737d-4805-981a-e102986f8479-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv\" (UID: \"7aa80757-737d-4805-981a-e102986f8479\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" Jul 09 18:59:47.024859 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.024822 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aa80757-737d-4805-981a-e102986f8479-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv\" (UID: \"7aa80757-737d-4805-981a-e102986f8479\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" Jul 09 18:59:47.025051 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.024872 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxwmz\" (UniqueName: \"kubernetes.io/projected/7aa80757-737d-4805-981a-e102986f8479-kube-api-access-mxwmz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv\" (UID: \"7aa80757-737d-4805-981a-e102986f8479\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" Jul 09 18:59:47.025051 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.024926 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aa80757-737d-4805-981a-e102986f8479-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv\" (UID: \"7aa80757-737d-4805-981a-e102986f8479\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" Jul 09 18:59:47.025283 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.025266 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aa80757-737d-4805-981a-e102986f8479-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv\" (UID: \"7aa80757-737d-4805-981a-e102986f8479\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" Jul 09 18:59:47.025327 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.025310 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aa80757-737d-4805-981a-e102986f8479-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv\" (UID: \"7aa80757-737d-4805-981a-e102986f8479\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" Jul 09 18:59:47.034667 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.034646 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxwmz\" (UniqueName: \"kubernetes.io/projected/7aa80757-737d-4805-981a-e102986f8479-kube-api-access-mxwmz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv\" (UID: \"7aa80757-737d-4805-981a-e102986f8479\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" Jul 09 18:59:47.131349 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.131259 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" Jul 09 18:59:47.260204 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.260178 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv"] Jul 09 18:59:47.262569 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:59:47.262540 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa80757_737d_4805_981a_e102986f8479.slice/crio-fe3d4e245739a4a0e723c3771959cc193a2c95a8c14e1229ac61ab2b76bbb124 WatchSource:0}: Error finding container fe3d4e245739a4a0e723c3771959cc193a2c95a8c14e1229ac61ab2b76bbb124: Status 404 returned error can't find the container with id fe3d4e245739a4a0e723c3771959cc193a2c95a8c14e1229ac61ab2b76bbb124 Jul 09 18:59:47.285687 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.285658 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" event={"ID":"7aa80757-737d-4805-981a-e102986f8479","Type":"ContainerStarted","Data":"fe3d4e245739a4a0e723c3771959cc193a2c95a8c14e1229ac61ab2b76bbb124"} Jul 09 18:59:47.620466 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.620435 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9"] Jul 09 18:59:47.622877 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.622861 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" Jul 09 18:59:47.634720 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.634688 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9"] Jul 09 18:59:47.731750 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.731713 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03a598a3-74c3-4360-946b-c8fbf548dcf6-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9\" (UID: \"03a598a3-74c3-4360-946b-c8fbf548dcf6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" Jul 09 18:59:47.731750 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.731751 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03a598a3-74c3-4360-946b-c8fbf548dcf6-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9\" (UID: \"03a598a3-74c3-4360-946b-c8fbf548dcf6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" Jul 09 18:59:47.732023 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.731853 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk8sx\" (UniqueName: \"kubernetes.io/projected/03a598a3-74c3-4360-946b-c8fbf548dcf6-kube-api-access-kk8sx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9\" (UID: \"03a598a3-74c3-4360-946b-c8fbf548dcf6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" Jul 09 18:59:47.832267 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.832231 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk8sx\" (UniqueName: \"kubernetes.io/projected/03a598a3-74c3-4360-946b-c8fbf548dcf6-kube-api-access-kk8sx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9\" (UID: \"03a598a3-74c3-4360-946b-c8fbf548dcf6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" Jul 09 18:59:47.832655 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.832286 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03a598a3-74c3-4360-946b-c8fbf548dcf6-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9\" (UID: \"03a598a3-74c3-4360-946b-c8fbf548dcf6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" Jul 09 18:59:47.832655 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.832470 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03a598a3-74c3-4360-946b-c8fbf548dcf6-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9\" (UID: \"03a598a3-74c3-4360-946b-c8fbf548dcf6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" Jul 09 18:59:47.832736 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.832653 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03a598a3-74c3-4360-946b-c8fbf548dcf6-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9\" (UID: \"03a598a3-74c3-4360-946b-c8fbf548dcf6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" Jul 09 18:59:47.832858 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.832837 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03a598a3-74c3-4360-946b-c8fbf548dcf6-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9\" (UID: \"03a598a3-74c3-4360-946b-c8fbf548dcf6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" Jul 09 18:59:47.842396 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.842371 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk8sx\" (UniqueName: \"kubernetes.io/projected/03a598a3-74c3-4360-946b-c8fbf548dcf6-kube-api-access-kk8sx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9\" (UID: \"03a598a3-74c3-4360-946b-c8fbf548dcf6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" Jul 09 18:59:47.931283 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:47.931207 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" Jul 09 18:59:48.059659 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.059633 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9"] Jul 09 18:59:48.061675 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:59:48.061650 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a598a3_74c3_4360_946b_c8fbf548dcf6.slice/crio-c80ea310b205b76ca2aeddadbcfef25d7cc4ec3c90521996d9c1fb9a81ff825c WatchSource:0}: Error finding container c80ea310b205b76ca2aeddadbcfef25d7cc4ec3c90521996d9c1fb9a81ff825c: Status 404 returned error can't find the container with id c80ea310b205b76ca2aeddadbcfef25d7cc4ec3c90521996d9c1fb9a81ff825c Jul 09 18:59:48.209459 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.209386 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh"] Jul 09 18:59:48.212079 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.212064 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" Jul 09 18:59:48.223375 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.223345 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh"] Jul 09 18:59:48.291566 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.291534 2569 generic.go:356] "Generic (PLEG): container finished" podID="03a598a3-74c3-4360-946b-c8fbf548dcf6" containerID="ff5e521f5839d05aadf38f12204462fe3044d7a2023a1dccad440c10141e2c2a" exitCode=0 Jul 09 18:59:48.291737 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.291624 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" event={"ID":"03a598a3-74c3-4360-946b-c8fbf548dcf6","Type":"ContainerDied","Data":"ff5e521f5839d05aadf38f12204462fe3044d7a2023a1dccad440c10141e2c2a"} Jul 09 18:59:48.291737 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.291670 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" event={"ID":"03a598a3-74c3-4360-946b-c8fbf548dcf6","Type":"ContainerStarted","Data":"c80ea310b205b76ca2aeddadbcfef25d7cc4ec3c90521996d9c1fb9a81ff825c"} Jul 09 18:59:48.293224 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.293203 2569 generic.go:356] "Generic (PLEG): container finished" podID="7aa80757-737d-4805-981a-e102986f8479" containerID="ce0741983067613c6a1f7051a98e10561a3eb28fe867454e9b06a3d3c5416533" exitCode=0 Jul 09 18:59:48.293369 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.293228 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" event={"ID":"7aa80757-737d-4805-981a-e102986f8479","Type":"ContainerDied","Data":"ce0741983067613c6a1f7051a98e10561a3eb28fe867454e9b06a3d3c5416533"} Jul 09 18:59:48.336442 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.336405 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh\" (UID: \"6bd73eca-bfc6-4d0a-950b-0875ff85f40b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" Jul 09 18:59:48.336593 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.336468 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5ctc\" (UniqueName: \"kubernetes.io/projected/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-kube-api-access-r5ctc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh\" (UID: \"6bd73eca-bfc6-4d0a-950b-0875ff85f40b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" Jul 09 18:59:48.336593 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.336567 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh\" (UID: \"6bd73eca-bfc6-4d0a-950b-0875ff85f40b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" Jul 09 18:59:48.437226 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.437195 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh\" (UID: \"6bd73eca-bfc6-4d0a-950b-0875ff85f40b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" Jul 09 18:59:48.437365 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.437246 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ctc\" (UniqueName: \"kubernetes.io/projected/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-kube-api-access-r5ctc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh\" (UID: \"6bd73eca-bfc6-4d0a-950b-0875ff85f40b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" Jul 09 18:59:48.437365 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.437294 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh\" (UID: \"6bd73eca-bfc6-4d0a-950b-0875ff85f40b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" Jul 09 18:59:48.437688 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.437670 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh\" (UID: \"6bd73eca-bfc6-4d0a-950b-0875ff85f40b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" Jul 09 18:59:48.437733 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.437682 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh\" (UID: \"6bd73eca-bfc6-4d0a-950b-0875ff85f40b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" Jul 09 18:59:48.445332 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.445303 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5ctc\" (UniqueName: \"kubernetes.io/projected/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-kube-api-access-r5ctc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh\" (UID: \"6bd73eca-bfc6-4d0a-950b-0875ff85f40b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" Jul 09 18:59:48.532909 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.532871 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" Jul 09 18:59:48.662629 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.662577 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh"] Jul 09 18:59:48.664723 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:59:48.664697 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd73eca_bfc6_4d0a_950b_0875ff85f40b.slice/crio-5a14364a38b05750c065aeacfaf30ef1eed2ccba8f9b17de56e6b75b6331e769 WatchSource:0}: Error finding container 5a14364a38b05750c065aeacfaf30ef1eed2ccba8f9b17de56e6b75b6331e769: Status 404 returned error can't find the container with id 5a14364a38b05750c065aeacfaf30ef1eed2ccba8f9b17de56e6b75b6331e769 Jul 09 18:59:48.810561 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.810487 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k"] Jul 09 18:59:48.813155 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.813140 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" Jul 09 18:59:48.824220 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.824194 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k"] Jul 09 18:59:48.942880 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.942846 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km2tq\" (UniqueName: \"kubernetes.io/projected/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-kube-api-access-km2tq\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k\" (UID: \"bb6716eb-53f8-48f9-a8c6-1484cd809ed1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" Jul 09 18:59:48.943305 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.942965 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k\" (UID: \"bb6716eb-53f8-48f9-a8c6-1484cd809ed1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" Jul 09 18:59:48.943305 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:48.943053 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k\" (UID: \"bb6716eb-53f8-48f9-a8c6-1484cd809ed1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" Jul 09 18:59:49.044145 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:49.044117 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k\" (UID: \"bb6716eb-53f8-48f9-a8c6-1484cd809ed1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" Jul 09 18:59:49.044344 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:49.044176 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km2tq\" (UniqueName: \"kubernetes.io/projected/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-kube-api-access-km2tq\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k\" (UID: \"bb6716eb-53f8-48f9-a8c6-1484cd809ed1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" Jul 09 18:59:49.044344 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:49.044255 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k\" (UID: \"bb6716eb-53f8-48f9-a8c6-1484cd809ed1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" Jul 09 18:59:49.044545 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:49.044525 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k\" (UID: \"bb6716eb-53f8-48f9-a8c6-1484cd809ed1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" Jul 09 18:59:49.044614 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:49.044549 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k\" (UID: \"bb6716eb-53f8-48f9-a8c6-1484cd809ed1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" Jul 09 18:59:49.052405 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:49.052382 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km2tq\" (UniqueName: \"kubernetes.io/projected/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-kube-api-access-km2tq\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k\" (UID: \"bb6716eb-53f8-48f9-a8c6-1484cd809ed1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" Jul 09 18:59:49.129430 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:49.129404 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" Jul 09 18:59:49.295431 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:49.295408 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k"] Jul 09 18:59:49.297387 ip-10-0-133-185 kubenswrapper[2569]: W0709 18:59:49.297361 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6716eb_53f8_48f9_a8c6_1484cd809ed1.slice/crio-38ef977bd7f269dcf0056e38f706f65667de5d9becc8a6c9602023e9c44f19c9 WatchSource:0}: Error finding container 38ef977bd7f269dcf0056e38f706f65667de5d9becc8a6c9602023e9c44f19c9: Status 404 returned error can't find the container with id 38ef977bd7f269dcf0056e38f706f65667de5d9becc8a6c9602023e9c44f19c9 Jul 09 18:59:49.300802 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:49.300782 2569 generic.go:356] "Generic (PLEG): container finished" podID="6bd73eca-bfc6-4d0a-950b-0875ff85f40b" containerID="cb00541185263d1ac0979135abac5253e298c4268ac5b3c3749c5d5781c8d172" exitCode=0 Jul 09 18:59:49.300890 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:49.300866 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" event={"ID":"6bd73eca-bfc6-4d0a-950b-0875ff85f40b","Type":"ContainerDied","Data":"cb00541185263d1ac0979135abac5253e298c4268ac5b3c3749c5d5781c8d172"} Jul 09 18:59:49.300982 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:49.300919 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" event={"ID":"6bd73eca-bfc6-4d0a-950b-0875ff85f40b","Type":"ContainerStarted","Data":"5a14364a38b05750c065aeacfaf30ef1eed2ccba8f9b17de56e6b75b6331e769"} Jul 09 18:59:49.302754 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:49.302733 2569 generic.go:356] "Generic (PLEG): container finished" podID="03a598a3-74c3-4360-946b-c8fbf548dcf6" containerID="118d1209201d131dd7060699ac2f7df8fbe0f6714074c13f0534da0574731e46" exitCode=0 Jul 09 18:59:49.302843 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:49.302811 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" event={"ID":"03a598a3-74c3-4360-946b-c8fbf548dcf6","Type":"ContainerDied","Data":"118d1209201d131dd7060699ac2f7df8fbe0f6714074c13f0534da0574731e46"} Jul 09 18:59:49.305857 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:49.305834 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" event={"ID":"7aa80757-737d-4805-981a-e102986f8479","Type":"ContainerStarted","Data":"bfdff447a40b7c56b97af82b8091e6d8971a267e64ffea295e3c1953c57fdf71"} Jul 09 18:59:50.313666 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:50.313640 2569 generic.go:356] "Generic (PLEG): container finished" podID="03a598a3-74c3-4360-946b-c8fbf548dcf6" containerID="76fe2940ce4bf43660fbde78ff0c4a14f7659b368bedc2ddfc9f2119bf63f29c" exitCode=0 Jul 09 18:59:50.314032 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:50.313679 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" event={"ID":"03a598a3-74c3-4360-946b-c8fbf548dcf6","Type":"ContainerDied","Data":"76fe2940ce4bf43660fbde78ff0c4a14f7659b368bedc2ddfc9f2119bf63f29c"} Jul 09 18:59:50.315413 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:50.315390 2569 generic.go:356] "Generic (PLEG): container finished" podID="7aa80757-737d-4805-981a-e102986f8479" containerID="bfdff447a40b7c56b97af82b8091e6d8971a267e64ffea295e3c1953c57fdf71" exitCode=0 Jul 09 18:59:50.315504 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:50.315460 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" event={"ID":"7aa80757-737d-4805-981a-e102986f8479","Type":"ContainerDied","Data":"bfdff447a40b7c56b97af82b8091e6d8971a267e64ffea295e3c1953c57fdf71"} Jul 09 18:59:50.317309 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:50.317289 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" event={"ID":"6bd73eca-bfc6-4d0a-950b-0875ff85f40b","Type":"ContainerStarted","Data":"54fb0ce2f39dbc587f357eb43af4ca32e45457cb0f58e8f073ef361fce36438e"} Jul 09 18:59:50.318812 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:50.318791 2569 generic.go:356] "Generic (PLEG): container finished" podID="bb6716eb-53f8-48f9-a8c6-1484cd809ed1" containerID="44b46eab4c249a616a8c35c2d8a9d7069e90c807a49e437e744acfe33bb902f3" exitCode=0 Jul 09 18:59:50.318882 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:50.318845 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" event={"ID":"bb6716eb-53f8-48f9-a8c6-1484cd809ed1","Type":"ContainerDied","Data":"44b46eab4c249a616a8c35c2d8a9d7069e90c807a49e437e744acfe33bb902f3"} Jul 09 18:59:50.318882 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:50.318875 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" event={"ID":"bb6716eb-53f8-48f9-a8c6-1484cd809ed1","Type":"ContainerStarted","Data":"38ef977bd7f269dcf0056e38f706f65667de5d9becc8a6c9602023e9c44f19c9"} Jul 09 18:59:51.325589 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.325505 2569 generic.go:356] "Generic (PLEG): container finished" podID="6bd73eca-bfc6-4d0a-950b-0875ff85f40b" containerID="54fb0ce2f39dbc587f357eb43af4ca32e45457cb0f58e8f073ef361fce36438e" exitCode=0 Jul 09 18:59:51.325589 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.325536 2569 generic.go:356] "Generic (PLEG): container finished" podID="6bd73eca-bfc6-4d0a-950b-0875ff85f40b" containerID="65800cf8c0ec3e1eef3a311531d4b8a1b0ef091923be3016d04d4e3e5afec792" exitCode=0 Jul 09 18:59:51.326039 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.325594 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" event={"ID":"6bd73eca-bfc6-4d0a-950b-0875ff85f40b","Type":"ContainerDied","Data":"54fb0ce2f39dbc587f357eb43af4ca32e45457cb0f58e8f073ef361fce36438e"} Jul 09 18:59:51.326039 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.325653 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" event={"ID":"6bd73eca-bfc6-4d0a-950b-0875ff85f40b","Type":"ContainerDied","Data":"65800cf8c0ec3e1eef3a311531d4b8a1b0ef091923be3016d04d4e3e5afec792"} Jul 09 18:59:51.327417 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.327397 2569 generic.go:356] "Generic (PLEG): container finished" podID="bb6716eb-53f8-48f9-a8c6-1484cd809ed1" containerID="b6c616829b060a889e6bfc33f0dc2d222182933f85b6bba8f30e68d8fcc57503" exitCode=0 Jul 09 18:59:51.327549 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.327419 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" event={"ID":"bb6716eb-53f8-48f9-a8c6-1484cd809ed1","Type":"ContainerDied","Data":"b6c616829b060a889e6bfc33f0dc2d222182933f85b6bba8f30e68d8fcc57503"} Jul 09 18:59:51.329684 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.329656 2569 generic.go:356] "Generic (PLEG): container finished" podID="7aa80757-737d-4805-981a-e102986f8479" containerID="6332d7e8d6008970cab8c6a345c085b0d223f2d9be2ae23977f245a0bcaf7537" exitCode=0 Jul 09 18:59:51.329781 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.329743 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" event={"ID":"7aa80757-737d-4805-981a-e102986f8479","Type":"ContainerDied","Data":"6332d7e8d6008970cab8c6a345c085b0d223f2d9be2ae23977f245a0bcaf7537"} Jul 09 18:59:51.463100 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.463074 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" Jul 09 18:59:51.567007 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.566968 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03a598a3-74c3-4360-946b-c8fbf548dcf6-bundle\") pod \"03a598a3-74c3-4360-946b-c8fbf548dcf6\" (UID: \"03a598a3-74c3-4360-946b-c8fbf548dcf6\") " Jul 09 18:59:51.567007 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.567012 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk8sx\" (UniqueName: \"kubernetes.io/projected/03a598a3-74c3-4360-946b-c8fbf548dcf6-kube-api-access-kk8sx\") pod \"03a598a3-74c3-4360-946b-c8fbf548dcf6\" (UID: \"03a598a3-74c3-4360-946b-c8fbf548dcf6\") " Jul 09 18:59:51.567225 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.567105 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03a598a3-74c3-4360-946b-c8fbf548dcf6-util\") pod \"03a598a3-74c3-4360-946b-c8fbf548dcf6\" (UID: \"03a598a3-74c3-4360-946b-c8fbf548dcf6\") " Jul 09 18:59:51.567553 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.567529 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a598a3-74c3-4360-946b-c8fbf548dcf6-bundle" (OuterVolumeSpecName: "bundle") pod "03a598a3-74c3-4360-946b-c8fbf548dcf6" (UID: "03a598a3-74c3-4360-946b-c8fbf548dcf6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:59:51.569303 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.569279 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a598a3-74c3-4360-946b-c8fbf548dcf6-kube-api-access-kk8sx" (OuterVolumeSpecName: "kube-api-access-kk8sx") pod "03a598a3-74c3-4360-946b-c8fbf548dcf6" (UID: "03a598a3-74c3-4360-946b-c8fbf548dcf6"). InnerVolumeSpecName "kube-api-access-kk8sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:59:51.572605 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.572571 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a598a3-74c3-4360-946b-c8fbf548dcf6-util" (OuterVolumeSpecName: "util") pod "03a598a3-74c3-4360-946b-c8fbf548dcf6" (UID: "03a598a3-74c3-4360-946b-c8fbf548dcf6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:59:51.667993 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.667893 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03a598a3-74c3-4360-946b-c8fbf548dcf6-util\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:59:51.667993 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.667925 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03a598a3-74c3-4360-946b-c8fbf548dcf6-bundle\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:59:51.667993 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:51.667937 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk8sx\" (UniqueName: \"kubernetes.io/projected/03a598a3-74c3-4360-946b-c8fbf548dcf6-kube-api-access-kk8sx\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:59:52.336834 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.336754 2569 generic.go:356] "Generic (PLEG): container finished" podID="bb6716eb-53f8-48f9-a8c6-1484cd809ed1" containerID="9c225a929c3c871fb5ef228f4a8d5e77abca121143b2b24f0976a6851b51e9f7" exitCode=0 Jul 09 18:59:52.337249 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.336850 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" event={"ID":"bb6716eb-53f8-48f9-a8c6-1484cd809ed1","Type":"ContainerDied","Data":"9c225a929c3c871fb5ef228f4a8d5e77abca121143b2b24f0976a6851b51e9f7"} Jul 09 18:59:52.338638 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.338618 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" event={"ID":"03a598a3-74c3-4360-946b-c8fbf548dcf6","Type":"ContainerDied","Data":"c80ea310b205b76ca2aeddadbcfef25d7cc4ec3c90521996d9c1fb9a81ff825c"} Jul 09 18:59:52.338691 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.338653 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9" Jul 09 18:59:52.338691 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.338654 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c80ea310b205b76ca2aeddadbcfef25d7cc4ec3c90521996d9c1fb9a81ff825c" Jul 09 18:59:52.491873 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.491851 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" Jul 09 18:59:52.494079 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.494062 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" Jul 09 18:59:52.677336 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.677246 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-util\") pod \"6bd73eca-bfc6-4d0a-950b-0875ff85f40b\" (UID: \"6bd73eca-bfc6-4d0a-950b-0875ff85f40b\") " Jul 09 18:59:52.685206 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.677657 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aa80757-737d-4805-981a-e102986f8479-util\") pod \"7aa80757-737d-4805-981a-e102986f8479\" (UID: \"7aa80757-737d-4805-981a-e102986f8479\") " Jul 09 18:59:52.685206 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.677708 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aa80757-737d-4805-981a-e102986f8479-bundle\") pod \"7aa80757-737d-4805-981a-e102986f8479\" (UID: \"7aa80757-737d-4805-981a-e102986f8479\") " Jul 09 18:59:52.685206 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.677763 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-bundle\") pod \"6bd73eca-bfc6-4d0a-950b-0875ff85f40b\" (UID: \"6bd73eca-bfc6-4d0a-950b-0875ff85f40b\") " Jul 09 18:59:52.685206 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.677807 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5ctc\" (UniqueName: \"kubernetes.io/projected/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-kube-api-access-r5ctc\") pod \"6bd73eca-bfc6-4d0a-950b-0875ff85f40b\" (UID: \"6bd73eca-bfc6-4d0a-950b-0875ff85f40b\") " Jul 09 18:59:52.685206 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.677983 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxwmz\" (UniqueName: \"kubernetes.io/projected/7aa80757-737d-4805-981a-e102986f8479-kube-api-access-mxwmz\") pod \"7aa80757-737d-4805-981a-e102986f8479\" (UID: \"7aa80757-737d-4805-981a-e102986f8479\") " Jul 09 18:59:52.685206 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.679261 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-bundle" (OuterVolumeSpecName: "bundle") pod "6bd73eca-bfc6-4d0a-950b-0875ff85f40b" (UID: "6bd73eca-bfc6-4d0a-950b-0875ff85f40b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:59:52.685206 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.679418 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa80757-737d-4805-981a-e102986f8479-bundle" (OuterVolumeSpecName: "bundle") pod "7aa80757-737d-4805-981a-e102986f8479" (UID: "7aa80757-737d-4805-981a-e102986f8479"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:59:52.685655 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.685612 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-kube-api-access-r5ctc" (OuterVolumeSpecName: "kube-api-access-r5ctc") pod "6bd73eca-bfc6-4d0a-950b-0875ff85f40b" (UID: "6bd73eca-bfc6-4d0a-950b-0875ff85f40b"). InnerVolumeSpecName "kube-api-access-r5ctc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:59:52.686286 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.686259 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa80757-737d-4805-981a-e102986f8479-util" (OuterVolumeSpecName: "util") pod "7aa80757-737d-4805-981a-e102986f8479" (UID: "7aa80757-737d-4805-981a-e102986f8479"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:59:52.686443 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.686419 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-util" (OuterVolumeSpecName: "util") pod "6bd73eca-bfc6-4d0a-950b-0875ff85f40b" (UID: "6bd73eca-bfc6-4d0a-950b-0875ff85f40b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:59:52.687272 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.687245 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa80757-737d-4805-981a-e102986f8479-kube-api-access-mxwmz" (OuterVolumeSpecName: "kube-api-access-mxwmz") pod "7aa80757-737d-4805-981a-e102986f8479" (UID: "7aa80757-737d-4805-981a-e102986f8479"). InnerVolumeSpecName "kube-api-access-mxwmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:59:52.779388 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.779353 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aa80757-737d-4805-981a-e102986f8479-util\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:59:52.779388 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.779385 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aa80757-737d-4805-981a-e102986f8479-bundle\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:59:52.779388 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.779395 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-bundle\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:59:52.779620 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.779407 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5ctc\" (UniqueName: \"kubernetes.io/projected/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-kube-api-access-r5ctc\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:59:52.779620 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.779422 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxwmz\" (UniqueName: \"kubernetes.io/projected/7aa80757-737d-4805-981a-e102986f8479-kube-api-access-mxwmz\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:59:52.779620 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:52.779432 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6bd73eca-bfc6-4d0a-950b-0875ff85f40b-util\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:59:53.345109 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.345022 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" Jul 09 18:59:53.345109 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.345038 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh" event={"ID":"6bd73eca-bfc6-4d0a-950b-0875ff85f40b","Type":"ContainerDied","Data":"5a14364a38b05750c065aeacfaf30ef1eed2ccba8f9b17de56e6b75b6331e769"} Jul 09 18:59:53.345109 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.345096 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a14364a38b05750c065aeacfaf30ef1eed2ccba8f9b17de56e6b75b6331e769" Jul 09 18:59:53.346791 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.346760 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" event={"ID":"7aa80757-737d-4805-981a-e102986f8479","Type":"ContainerDied","Data":"fe3d4e245739a4a0e723c3771959cc193a2c95a8c14e1229ac61ab2b76bbb124"} Jul 09 18:59:53.346791 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.346789 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv" Jul 09 18:59:53.346970 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.346805 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe3d4e245739a4a0e723c3771959cc193a2c95a8c14e1229ac61ab2b76bbb124" Jul 09 18:59:53.467916 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.467893 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" Jul 09 18:59:53.586592 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.586551 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-bundle\") pod \"bb6716eb-53f8-48f9-a8c6-1484cd809ed1\" (UID: \"bb6716eb-53f8-48f9-a8c6-1484cd809ed1\") " Jul 09 18:59:53.586592 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.586589 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-util\") pod \"bb6716eb-53f8-48f9-a8c6-1484cd809ed1\" (UID: \"bb6716eb-53f8-48f9-a8c6-1484cd809ed1\") " Jul 09 18:59:53.586801 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.586611 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km2tq\" (UniqueName: \"kubernetes.io/projected/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-kube-api-access-km2tq\") pod \"bb6716eb-53f8-48f9-a8c6-1484cd809ed1\" (UID: \"bb6716eb-53f8-48f9-a8c6-1484cd809ed1\") " Jul 09 18:59:53.587090 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.587067 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-bundle" (OuterVolumeSpecName: "bundle") pod "bb6716eb-53f8-48f9-a8c6-1484cd809ed1" (UID: "bb6716eb-53f8-48f9-a8c6-1484cd809ed1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:59:53.588697 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.588671 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-kube-api-access-km2tq" (OuterVolumeSpecName: "kube-api-access-km2tq") pod "bb6716eb-53f8-48f9-a8c6-1484cd809ed1" (UID: "bb6716eb-53f8-48f9-a8c6-1484cd809ed1"). InnerVolumeSpecName "kube-api-access-km2tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:59:53.592077 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.592059 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-util" (OuterVolumeSpecName: "util") pod "bb6716eb-53f8-48f9-a8c6-1484cd809ed1" (UID: "bb6716eb-53f8-48f9-a8c6-1484cd809ed1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:59:53.687796 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.687719 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-bundle\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:59:53.687796 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.687746 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-util\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:59:53.687796 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:53.687755 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km2tq\" (UniqueName: \"kubernetes.io/projected/bb6716eb-53f8-48f9-a8c6-1484cd809ed1-kube-api-access-km2tq\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 18:59:54.355204 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:54.355169 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" event={"ID":"bb6716eb-53f8-48f9-a8c6-1484cd809ed1","Type":"ContainerDied","Data":"38ef977bd7f269dcf0056e38f706f65667de5d9becc8a6c9602023e9c44f19c9"} Jul 09 18:59:54.355204 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:54.355196 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k" Jul 09 18:59:54.355621 ip-10-0-133-185 kubenswrapper[2569]: I0709 18:59:54.355223 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38ef977bd7f269dcf0056e38f706f65667de5d9becc8a6c9602023e9c44f19c9" Jul 09 19:00:11.338512 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:11.338466 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d5567bcd7-hzlnv"] Jul 09 19:00:22.168194 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168155 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz"] Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168453 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa80757-737d-4805-981a-e102986f8479" containerName="pull" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168463 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa80757-737d-4805-981a-e102986f8479" containerName="pull" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168474 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa80757-737d-4805-981a-e102986f8479" containerName="extract" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168481 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa80757-737d-4805-981a-e102986f8479" containerName="extract" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168489 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd73eca-bfc6-4d0a-950b-0875ff85f40b" containerName="extract" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168495 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd73eca-bfc6-4d0a-950b-0875ff85f40b" containerName="extract" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168502 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a598a3-74c3-4360-946b-c8fbf548dcf6" containerName="util" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168506 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a598a3-74c3-4360-946b-c8fbf548dcf6" containerName="util" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168513 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa80757-737d-4805-981a-e102986f8479" containerName="util" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168519 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa80757-737d-4805-981a-e102986f8479" containerName="util" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168524 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb6716eb-53f8-48f9-a8c6-1484cd809ed1" containerName="extract" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168528 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb6716eb-53f8-48f9-a8c6-1484cd809ed1" containerName="extract" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168537 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb6716eb-53f8-48f9-a8c6-1484cd809ed1" containerName="util" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168542 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb6716eb-53f8-48f9-a8c6-1484cd809ed1" containerName="util" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168548 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a598a3-74c3-4360-946b-c8fbf548dcf6" containerName="extract" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168553 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a598a3-74c3-4360-946b-c8fbf548dcf6" containerName="extract" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168560 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb6716eb-53f8-48f9-a8c6-1484cd809ed1" containerName="pull" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168564 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb6716eb-53f8-48f9-a8c6-1484cd809ed1" containerName="pull" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168572 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd73eca-bfc6-4d0a-950b-0875ff85f40b" containerName="util" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168576 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd73eca-bfc6-4d0a-950b-0875ff85f40b" containerName="util" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168584 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd73eca-bfc6-4d0a-950b-0875ff85f40b" containerName="pull" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168589 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd73eca-bfc6-4d0a-950b-0875ff85f40b" containerName="pull" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168595 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a598a3-74c3-4360-946b-c8fbf548dcf6" containerName="pull" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168600 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a598a3-74c3-4360-946b-c8fbf548dcf6" containerName="pull" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168644 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb6716eb-53f8-48f9-a8c6-1484cd809ed1" containerName="extract" Jul 09 19:00:22.168656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168652 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd73eca-bfc6-4d0a-950b-0875ff85f40b" containerName="extract" Jul 09 19:00:22.169511 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168676 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a598a3-74c3-4360-946b-c8fbf548dcf6" containerName="extract" Jul 09 19:00:22.169511 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.168683 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa80757-737d-4805-981a-e102986f8479" containerName="extract" Jul 09 19:00:22.170860 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.170844 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz" Jul 09 19:00:22.175090 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.175069 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 19:00:22.175220 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.175169 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 19:00:22.175220 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.175178 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 09 19:00:22.175678 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.175661 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 09 19:00:22.176421 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.176404 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xcftl\"" Jul 09 19:00:22.186704 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.186685 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz"] Jul 09 19:00:22.229258 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.229226 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkfwh\" (UniqueName: \"kubernetes.io/projected/21b026d8-5239-4098-b1e7-3cdcf6f07bc1-kube-api-access-bkfwh\") pod \"kuadrant-console-plugin-66b5774985-kvflz\" (UID: \"21b026d8-5239-4098-b1e7-3cdcf6f07bc1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz" Jul 09 19:00:22.229436 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.229334 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/21b026d8-5239-4098-b1e7-3cdcf6f07bc1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kvflz\" (UID: \"21b026d8-5239-4098-b1e7-3cdcf6f07bc1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz" Jul 09 19:00:22.229436 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.229394 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/21b026d8-5239-4098-b1e7-3cdcf6f07bc1-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kvflz\" (UID: \"21b026d8-5239-4098-b1e7-3cdcf6f07bc1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz" Jul 09 19:00:22.330025 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.329992 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/21b026d8-5239-4098-b1e7-3cdcf6f07bc1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kvflz\" (UID: \"21b026d8-5239-4098-b1e7-3cdcf6f07bc1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz" Jul 09 19:00:22.330201 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.330043 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/21b026d8-5239-4098-b1e7-3cdcf6f07bc1-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kvflz\" (UID: \"21b026d8-5239-4098-b1e7-3cdcf6f07bc1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz" Jul 09 19:00:22.330201 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.330062 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkfwh\" (UniqueName: \"kubernetes.io/projected/21b026d8-5239-4098-b1e7-3cdcf6f07bc1-kube-api-access-bkfwh\") pod \"kuadrant-console-plugin-66b5774985-kvflz\" (UID: \"21b026d8-5239-4098-b1e7-3cdcf6f07bc1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz" Jul 09 19:00:22.330201 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:00:22.330160 2569 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 09 19:00:22.330317 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:00:22.330242 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b026d8-5239-4098-b1e7-3cdcf6f07bc1-plugin-serving-cert podName:21b026d8-5239-4098-b1e7-3cdcf6f07bc1 nodeName:}" failed. No retries permitted until 2026-07-09 19:00:22.830219002 +0000 UTC m=+407.655382028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/21b026d8-5239-4098-b1e7-3cdcf6f07bc1-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-kvflz" (UID: "21b026d8-5239-4098-b1e7-3cdcf6f07bc1") : secret "plugin-serving-cert" not found Jul 09 19:00:22.330803 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.330786 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/21b026d8-5239-4098-b1e7-3cdcf6f07bc1-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kvflz\" (UID: \"21b026d8-5239-4098-b1e7-3cdcf6f07bc1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz" Jul 09 19:00:22.344517 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.344481 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkfwh\" (UniqueName: \"kubernetes.io/projected/21b026d8-5239-4098-b1e7-3cdcf6f07bc1-kube-api-access-bkfwh\") pod \"kuadrant-console-plugin-66b5774985-kvflz\" (UID: \"21b026d8-5239-4098-b1e7-3cdcf6f07bc1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz" Jul 09 19:00:22.833266 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.833231 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/21b026d8-5239-4098-b1e7-3cdcf6f07bc1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kvflz\" (UID: \"21b026d8-5239-4098-b1e7-3cdcf6f07bc1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz" Jul 09 19:00:22.835666 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:22.835643 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/21b026d8-5239-4098-b1e7-3cdcf6f07bc1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kvflz\" (UID: \"21b026d8-5239-4098-b1e7-3cdcf6f07bc1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz" Jul 09 19:00:23.079877 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:23.079843 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz" Jul 09 19:00:23.208166 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:23.208142 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz"] Jul 09 19:00:23.210375 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:00:23.210352 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21b026d8_5239_4098_b1e7_3cdcf6f07bc1.slice/crio-2ecc3ec05c2b928d4aa8538e356e29ba06daf0237a39084d626155d3dff0cf19 WatchSource:0}: Error finding container 2ecc3ec05c2b928d4aa8538e356e29ba06daf0237a39084d626155d3dff0cf19: Status 404 returned error can't find the container with id 2ecc3ec05c2b928d4aa8538e356e29ba06daf0237a39084d626155d3dff0cf19 Jul 09 19:00:23.490016 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:23.489987 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz" event={"ID":"21b026d8-5239-4098-b1e7-3cdcf6f07bc1","Type":"ContainerStarted","Data":"2ecc3ec05c2b928d4aa8538e356e29ba06daf0237a39084d626155d3dff0cf19"} Jul 09 19:00:31.678187 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:31.678148 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj"] Jul 09 19:00:31.685233 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:31.685205 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:31.688033 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:31.688006 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zq2wt\"" Jul 09 19:00:31.699695 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:31.699671 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj"] Jul 09 19:00:31.815824 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:31.815786 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c56a1696-1723-4901-a94e-67c8b2110be6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-trlwj\" (UID: \"c56a1696-1723-4901-a94e-67c8b2110be6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:31.815824 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:31.815827 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7rtt\" (UniqueName: \"kubernetes.io/projected/c56a1696-1723-4901-a94e-67c8b2110be6-kube-api-access-c7rtt\") pod \"kuadrant-operator-controller-manager-55467d9f7-trlwj\" (UID: \"c56a1696-1723-4901-a94e-67c8b2110be6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:31.917394 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:31.917352 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c56a1696-1723-4901-a94e-67c8b2110be6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-trlwj\" (UID: \"c56a1696-1723-4901-a94e-67c8b2110be6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:31.917394 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:31.917401 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7rtt\" (UniqueName: \"kubernetes.io/projected/c56a1696-1723-4901-a94e-67c8b2110be6-kube-api-access-c7rtt\") pod \"kuadrant-operator-controller-manager-55467d9f7-trlwj\" (UID: \"c56a1696-1723-4901-a94e-67c8b2110be6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:31.917869 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:31.917839 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c56a1696-1723-4901-a94e-67c8b2110be6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-trlwj\" (UID: \"c56a1696-1723-4901-a94e-67c8b2110be6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:31.926668 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:31.926639 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7rtt\" (UniqueName: \"kubernetes.io/projected/c56a1696-1723-4901-a94e-67c8b2110be6-kube-api-access-c7rtt\") pod \"kuadrant-operator-controller-manager-55467d9f7-trlwj\" (UID: \"c56a1696-1723-4901-a94e-67c8b2110be6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:31.999552 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:31.999518 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:32.087043 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:32.087009 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj"] Jul 09 19:00:32.092370 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:32.092341 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj"] Jul 09 19:00:32.117229 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:32.117194 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f"] Jul 09 19:00:32.122671 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:32.122645 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" Jul 09 19:00:32.136004 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:32.135970 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f"] Jul 09 19:00:32.221003 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:32.220897 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t7vv\" (UniqueName: \"kubernetes.io/projected/f0f4649a-143e-46f5-8594-96696002fe7f-kube-api-access-9t7vv\") pod \"kuadrant-operator-controller-manager-55467d9f7-c5n4f\" (UID: \"f0f4649a-143e-46f5-8594-96696002fe7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" Jul 09 19:00:32.221196 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:32.221066 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0f4649a-143e-46f5-8594-96696002fe7f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-c5n4f\" (UID: \"f0f4649a-143e-46f5-8594-96696002fe7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" Jul 09 19:00:32.322184 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:32.322109 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0f4649a-143e-46f5-8594-96696002fe7f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-c5n4f\" (UID: \"f0f4649a-143e-46f5-8594-96696002fe7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" Jul 09 19:00:32.322184 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:32.322168 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t7vv\" (UniqueName: \"kubernetes.io/projected/f0f4649a-143e-46f5-8594-96696002fe7f-kube-api-access-9t7vv\") pod \"kuadrant-operator-controller-manager-55467d9f7-c5n4f\" (UID: \"f0f4649a-143e-46f5-8594-96696002fe7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" Jul 09 19:00:32.322545 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:32.322520 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0f4649a-143e-46f5-8594-96696002fe7f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-c5n4f\" (UID: \"f0f4649a-143e-46f5-8594-96696002fe7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" Jul 09 19:00:32.331372 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:32.331345 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t7vv\" (UniqueName: \"kubernetes.io/projected/f0f4649a-143e-46f5-8594-96696002fe7f-kube-api-access-9t7vv\") pod \"kuadrant-operator-controller-manager-55467d9f7-c5n4f\" (UID: \"f0f4649a-143e-46f5-8594-96696002fe7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" Jul 09 19:00:32.435551 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:32.435520 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" Jul 09 19:00:36.359350 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:36.359287 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d5567bcd7-hzlnv" podUID="a54ec556-a025-4a3e-8fff-7de0cd201187" containerName="console" containerID="cri-o://dbbd03cd93ae1daf4c13bfdb0eca709d2f9490a444fb70254ea576fb028db36f" gracePeriod=15 Jul 09 19:00:43.433595 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:43.433548 2569 patch_prober.go:28] interesting pod/console-5d5567bcd7-hzlnv container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused" start-of-body= Jul 09 19:00:43.434075 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:43.433626 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-5d5567bcd7-hzlnv" podUID="a54ec556-a025-4a3e-8fff-7de0cd201187" containerName="console" probeResult="failure" output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused" Jul 09 19:00:46.209905 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.209077 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f"] Jul 09 19:00:46.214738 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:00:46.214709 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f4649a_143e_46f5_8594_96696002fe7f.slice/crio-137f01beb64244ec7fdba6db34b13e8b1b2f29583d296ca5b5fa867283062eee WatchSource:0}: Error finding container 137f01beb64244ec7fdba6db34b13e8b1b2f29583d296ca5b5fa867283062eee: Status 404 returned error can't find the container with id 137f01beb64244ec7fdba6db34b13e8b1b2f29583d296ca5b5fa867283062eee Jul 09 19:00:46.244714 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.244690 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 19:00:46.345973 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.345880 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-trusted-ca-bundle\") pod \"a54ec556-a025-4a3e-8fff-7de0cd201187\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " Jul 09 19:00:46.345973 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.345939 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-686k9\" (UniqueName: \"kubernetes.io/projected/a54ec556-a025-4a3e-8fff-7de0cd201187-kube-api-access-686k9\") pod \"a54ec556-a025-4a3e-8fff-7de0cd201187\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " Jul 09 19:00:46.346150 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.346123 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a54ec556-a025-4a3e-8fff-7de0cd201187-console-serving-cert\") pod \"a54ec556-a025-4a3e-8fff-7de0cd201187\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " Jul 09 19:00:46.346186 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.346160 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-oauth-serving-cert\") pod \"a54ec556-a025-4a3e-8fff-7de0cd201187\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " Jul 09 19:00:46.346227 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.346188 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a54ec556-a025-4a3e-8fff-7de0cd201187-console-oauth-config\") pod \"a54ec556-a025-4a3e-8fff-7de0cd201187\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " Jul 09 19:00:46.346227 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.346217 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-console-config\") pod \"a54ec556-a025-4a3e-8fff-7de0cd201187\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " Jul 09 19:00:46.346313 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.346273 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-service-ca\") pod \"a54ec556-a025-4a3e-8fff-7de0cd201187\" (UID: \"a54ec556-a025-4a3e-8fff-7de0cd201187\") " Jul 09 19:00:46.346313 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.346293 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a54ec556-a025-4a3e-8fff-7de0cd201187" (UID: "a54ec556-a025-4a3e-8fff-7de0cd201187"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:00:46.346483 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.346458 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a54ec556-a025-4a3e-8fff-7de0cd201187" (UID: "a54ec556-a025-4a3e-8fff-7de0cd201187"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:00:46.346577 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.346543 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-trusted-ca-bundle\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:00:46.346577 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.346562 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-oauth-serving-cert\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:00:46.346665 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.346639 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-console-config" (OuterVolumeSpecName: "console-config") pod "a54ec556-a025-4a3e-8fff-7de0cd201187" (UID: "a54ec556-a025-4a3e-8fff-7de0cd201187"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:00:46.346703 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.346669 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-service-ca" (OuterVolumeSpecName: "service-ca") pod "a54ec556-a025-4a3e-8fff-7de0cd201187" (UID: "a54ec556-a025-4a3e-8fff-7de0cd201187"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:00:46.348131 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.348111 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54ec556-a025-4a3e-8fff-7de0cd201187-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a54ec556-a025-4a3e-8fff-7de0cd201187" (UID: "a54ec556-a025-4a3e-8fff-7de0cd201187"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:00:46.348513 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.348493 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54ec556-a025-4a3e-8fff-7de0cd201187-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a54ec556-a025-4a3e-8fff-7de0cd201187" (UID: "a54ec556-a025-4a3e-8fff-7de0cd201187"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:00:46.348555 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.348496 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54ec556-a025-4a3e-8fff-7de0cd201187-kube-api-access-686k9" (OuterVolumeSpecName: "kube-api-access-686k9") pod "a54ec556-a025-4a3e-8fff-7de0cd201187" (UID: "a54ec556-a025-4a3e-8fff-7de0cd201187"). InnerVolumeSpecName "kube-api-access-686k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:00:46.447267 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.447234 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-686k9\" (UniqueName: \"kubernetes.io/projected/a54ec556-a025-4a3e-8fff-7de0cd201187-kube-api-access-686k9\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:00:46.447267 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.447258 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a54ec556-a025-4a3e-8fff-7de0cd201187-console-serving-cert\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:00:46.447267 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.447269 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a54ec556-a025-4a3e-8fff-7de0cd201187-console-oauth-config\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:00:46.447487 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.447280 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-console-config\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:00:46.447487 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.447288 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a54ec556-a025-4a3e-8fff-7de0cd201187-service-ca\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:00:46.619317 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.619228 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" event={"ID":"f0f4649a-143e-46f5-8594-96696002fe7f","Type":"ContainerStarted","Data":"137f01beb64244ec7fdba6db34b13e8b1b2f29583d296ca5b5fa867283062eee"} Jul 09 19:00:46.620500 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.620454 2569 generic.go:356] "Generic (PLEG): container finished" podID="a54ec556-a025-4a3e-8fff-7de0cd201187" containerID="dbbd03cd93ae1daf4c13bfdb0eca709d2f9490a444fb70254ea576fb028db36f" exitCode=0 Jul 09 19:00:46.620500 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.620493 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5567bcd7-hzlnv" Jul 09 19:00:46.620678 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.620520 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5567bcd7-hzlnv" event={"ID":"a54ec556-a025-4a3e-8fff-7de0cd201187","Type":"ContainerDied","Data":"dbbd03cd93ae1daf4c13bfdb0eca709d2f9490a444fb70254ea576fb028db36f"} Jul 09 19:00:46.620678 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.620562 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5567bcd7-hzlnv" event={"ID":"a54ec556-a025-4a3e-8fff-7de0cd201187","Type":"ContainerDied","Data":"0a9d028f5315ba589efa161feeb810d3b7efd9eb350f1f129f25191059a2619e"} Jul 09 19:00:46.620678 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.620585 2569 scope.go:117] "RemoveContainer" containerID="dbbd03cd93ae1daf4c13bfdb0eca709d2f9490a444fb70254ea576fb028db36f" Jul 09 19:00:46.622276 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.622253 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz" event={"ID":"21b026d8-5239-4098-b1e7-3cdcf6f07bc1","Type":"ContainerStarted","Data":"55453a65ac8fb36502d1316063e8560c4accb640febcae2de5e02ddf2c5081d8"} Jul 09 19:00:46.630273 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.630255 2569 scope.go:117] "RemoveContainer" containerID="dbbd03cd93ae1daf4c13bfdb0eca709d2f9490a444fb70254ea576fb028db36f" Jul 09 19:00:46.630503 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:00:46.630487 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbd03cd93ae1daf4c13bfdb0eca709d2f9490a444fb70254ea576fb028db36f\": container with ID starting with dbbd03cd93ae1daf4c13bfdb0eca709d2f9490a444fb70254ea576fb028db36f not found: ID does not exist" containerID="dbbd03cd93ae1daf4c13bfdb0eca709d2f9490a444fb70254ea576fb028db36f" Jul 09 19:00:46.630564 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.630510 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbd03cd93ae1daf4c13bfdb0eca709d2f9490a444fb70254ea576fb028db36f"} err="failed to get container status \"dbbd03cd93ae1daf4c13bfdb0eca709d2f9490a444fb70254ea576fb028db36f\": rpc error: code = NotFound desc = could not find container \"dbbd03cd93ae1daf4c13bfdb0eca709d2f9490a444fb70254ea576fb028db36f\": container with ID starting with dbbd03cd93ae1daf4c13bfdb0eca709d2f9490a444fb70254ea576fb028db36f not found: ID does not exist" Jul 09 19:00:46.647656 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.647604 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kvflz" podStartSLOduration=1.625122476 podStartE2EDuration="24.647590983s" totalImagesPullingTime="23.022468507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:46.645169681 +0000 UTC m=+431.470332717" watchObservedRunningTime="2026-07-09 19:00:46.647590983 +0000 UTC m=+431.472754020" Jul 09 19:00:46.649289 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:00:46.649263 2569 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54ec556_a025_4a3e_8fff_7de0cd201187.slice/crio-0a9d028f5315ba589efa161feeb810d3b7efd9eb350f1f129f25191059a2619e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54ec556_a025_4a3e_8fff_7de0cd201187.slice\": RecentStats: unable to find data in memory cache]" Jul 09 19:00:46.660872 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.660848 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d5567bcd7-hzlnv"] Jul 09 19:00:46.667714 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:46.667687 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d5567bcd7-hzlnv"] Jul 09 19:00:47.694877 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:47.694842 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a54ec556-a025-4a3e-8fff-7de0cd201187" path="/var/lib/kubelet/pods/a54ec556-a025-4a3e-8fff-7de0cd201187/volumes" Jul 09 19:00:48.733173 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:00:48.733121 2569 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-trlwj_kuadrant-system_c56a1696-1723-4901-a94e-67c8b2110be6_0(3873729ebd235fc14dfd4ef078ab96377ad00800fd54f2e55e8ddcc92275a840): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-trlwj to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 09 19:00:48.733496 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:00:48.733204 2569 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-trlwj_kuadrant-system_c56a1696-1723-4901-a94e-67c8b2110be6_0(3873729ebd235fc14dfd4ef078ab96377ad00800fd54f2e55e8ddcc92275a840): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-trlwj to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:49.643829 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:49.643800 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:49.646149 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:00:49.646122 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-trlwj\" is forbidden: User \"system:node:ip-10-0-133-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-185.ec2.internal' and this object" podUID="c56a1696-1723-4901-a94e-67c8b2110be6" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:49.661395 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:49.661367 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:49.663563 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:00:49.663538 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-trlwj\" is forbidden: User \"system:node:ip-10-0-133-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-185.ec2.internal' and this object" podUID="c56a1696-1723-4901-a94e-67c8b2110be6" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:49.676938 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:49.676905 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c56a1696-1723-4901-a94e-67c8b2110be6-extensions-socket-volume\") pod \"c56a1696-1723-4901-a94e-67c8b2110be6\" (UID: \"c56a1696-1723-4901-a94e-67c8b2110be6\") " Jul 09 19:00:49.677045 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:49.677008 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7rtt\" (UniqueName: \"kubernetes.io/projected/c56a1696-1723-4901-a94e-67c8b2110be6-kube-api-access-c7rtt\") pod \"c56a1696-1723-4901-a94e-67c8b2110be6\" (UID: \"c56a1696-1723-4901-a94e-67c8b2110be6\") " Jul 09 19:00:49.677168 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:49.677147 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56a1696-1723-4901-a94e-67c8b2110be6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c56a1696-1723-4901-a94e-67c8b2110be6" (UID: "c56a1696-1723-4901-a94e-67c8b2110be6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:00:49.678925 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:49.678904 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56a1696-1723-4901-a94e-67c8b2110be6-kube-api-access-c7rtt" (OuterVolumeSpecName: "kube-api-access-c7rtt") pod "c56a1696-1723-4901-a94e-67c8b2110be6" (UID: "c56a1696-1723-4901-a94e-67c8b2110be6"). InnerVolumeSpecName "kube-api-access-c7rtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:00:49.695335 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:49.695309 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c56a1696-1723-4901-a94e-67c8b2110be6" path="/var/lib/kubelet/pods/c56a1696-1723-4901-a94e-67c8b2110be6/volumes" Jul 09 19:00:49.778361 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:49.778144 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c56a1696-1723-4901-a94e-67c8b2110be6-extensions-socket-volume\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:00:49.778361 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:49.778176 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7rtt\" (UniqueName: \"kubernetes.io/projected/c56a1696-1723-4901-a94e-67c8b2110be6-kube-api-access-c7rtt\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:00:50.649555 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:50.649521 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:50.651910 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:00:50.651873 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-trlwj\" is forbidden: User \"system:node:ip-10-0-133-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-185.ec2.internal' and this object" podUID="c56a1696-1723-4901-a94e-67c8b2110be6" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:50.654067 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:00:50.654040 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-trlwj\" is forbidden: User \"system:node:ip-10-0-133-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-185.ec2.internal' and this object" podUID="c56a1696-1723-4901-a94e-67c8b2110be6" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-trlwj" Jul 09 19:00:52.661783 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:52.661738 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" event={"ID":"f0f4649a-143e-46f5-8594-96696002fe7f","Type":"ContainerStarted","Data":"78774cd5d204d21568aa86969938f16bc80a6a2370fde9230b0fcf3ce90f3bc1"} Jul 09 19:00:52.662185 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:52.661882 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" Jul 09 19:00:52.683854 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:00:52.683799 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" podStartSLOduration=15.098503449 podStartE2EDuration="20.683780771s" totalImagesPullingTime="5.585277322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:52.681016148 +0000 UTC m=+437.506179178" watchObservedRunningTime="2026-07-09 19:00:52.683780771 +0000 UTC m=+437.508943813" Jul 09 19:01:03.669749 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:03.669664 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" Jul 09 19:01:16.625423 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:16.625386 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f"] Jul 09 19:01:16.625903 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:16.625587 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" podUID="f0f4649a-143e-46f5-8594-96696002fe7f" containerName="manager" containerID="cri-o://78774cd5d204d21568aa86969938f16bc80a6a2370fde9230b0fcf3ce90f3bc1" gracePeriod=10 Jul 09 19:01:16.773464 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:16.773422 2569 generic.go:356] "Generic (PLEG): container finished" podID="f0f4649a-143e-46f5-8594-96696002fe7f" containerID="78774cd5d204d21568aa86969938f16bc80a6a2370fde9230b0fcf3ce90f3bc1" exitCode=0 Jul 09 19:01:16.773620 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:16.773501 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" event={"ID":"f0f4649a-143e-46f5-8594-96696002fe7f","Type":"ContainerDied","Data":"78774cd5d204d21568aa86969938f16bc80a6a2370fde9230b0fcf3ce90f3bc1"} Jul 09 19:01:16.892605 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:16.892547 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" Jul 09 19:01:17.017597 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:17.017565 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t7vv\" (UniqueName: \"kubernetes.io/projected/f0f4649a-143e-46f5-8594-96696002fe7f-kube-api-access-9t7vv\") pod \"f0f4649a-143e-46f5-8594-96696002fe7f\" (UID: \"f0f4649a-143e-46f5-8594-96696002fe7f\") " Jul 09 19:01:17.017776 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:17.017614 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0f4649a-143e-46f5-8594-96696002fe7f-extensions-socket-volume\") pod \"f0f4649a-143e-46f5-8594-96696002fe7f\" (UID: \"f0f4649a-143e-46f5-8594-96696002fe7f\") " Jul 09 19:01:17.018161 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:17.018133 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f4649a-143e-46f5-8594-96696002fe7f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f0f4649a-143e-46f5-8594-96696002fe7f" (UID: "f0f4649a-143e-46f5-8594-96696002fe7f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:01:17.019619 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:17.019591 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f4649a-143e-46f5-8594-96696002fe7f-kube-api-access-9t7vv" (OuterVolumeSpecName: "kube-api-access-9t7vv") pod "f0f4649a-143e-46f5-8594-96696002fe7f" (UID: "f0f4649a-143e-46f5-8594-96696002fe7f"). InnerVolumeSpecName "kube-api-access-9t7vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:01:17.118307 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:17.118270 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t7vv\" (UniqueName: \"kubernetes.io/projected/f0f4649a-143e-46f5-8594-96696002fe7f-kube-api-access-9t7vv\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:01:17.118307 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:17.118300 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0f4649a-143e-46f5-8594-96696002fe7f-extensions-socket-volume\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:01:17.780144 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:17.780115 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" Jul 09 19:01:17.780524 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:17.780116 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f" event={"ID":"f0f4649a-143e-46f5-8594-96696002fe7f","Type":"ContainerDied","Data":"137f01beb64244ec7fdba6db34b13e8b1b2f29583d296ca5b5fa867283062eee"} Jul 09 19:01:17.780524 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:17.780248 2569 scope.go:117] "RemoveContainer" containerID="78774cd5d204d21568aa86969938f16bc80a6a2370fde9230b0fcf3ce90f3bc1" Jul 09 19:01:17.799778 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:17.799755 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f"] Jul 09 19:01:17.806443 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:17.806422 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-c5n4f"] Jul 09 19:01:19.693960 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:19.693915 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0f4649a-143e-46f5-8594-96696002fe7f" path="/var/lib/kubelet/pods/f0f4649a-143e-46f5-8594-96696002fe7f/volumes" Jul 09 19:01:22.954918 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:22.954886 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-f9pdh"] Jul 09 19:01:22.955325 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:22.955215 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f4649a-143e-46f5-8594-96696002fe7f" containerName="manager" Jul 09 19:01:22.955325 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:22.955228 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f4649a-143e-46f5-8594-96696002fe7f" containerName="manager" Jul 09 19:01:22.955325 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:22.955246 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a54ec556-a025-4a3e-8fff-7de0cd201187" containerName="console" Jul 09 19:01:22.955325 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:22.955251 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a54ec556-a025-4a3e-8fff-7de0cd201187" containerName="console" Jul 09 19:01:22.955325 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:22.955302 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0f4649a-143e-46f5-8594-96696002fe7f" containerName="manager" Jul 09 19:01:22.955325 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:22.955309 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a54ec556-a025-4a3e-8fff-7de0cd201187" containerName="console" Jul 09 19:01:22.972419 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:22.972390 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-f9pdh"] Jul 09 19:01:22.972555 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:22.972500 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-f9pdh" Jul 09 19:01:22.974931 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:22.974908 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6tmgx\"" Jul 09 19:01:23.074265 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:23.074232 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76lzx\" (UniqueName: \"kubernetes.io/projected/6222f410-20d7-4a43-95e7-6d384d673e37-kube-api-access-76lzx\") pod \"authorino-58cd955675-f9pdh\" (UID: \"6222f410-20d7-4a43-95e7-6d384d673e37\") " pod="kuadrant-system/authorino-58cd955675-f9pdh" Jul 09 19:01:23.106115 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:23.106083 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-57gpf"] Jul 09 19:01:23.110625 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:23.110609 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-57gpf" Jul 09 19:01:23.117293 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:23.117255 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-57gpf"] Jul 09 19:01:23.175607 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:23.175571 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76lzx\" (UniqueName: \"kubernetes.io/projected/6222f410-20d7-4a43-95e7-6d384d673e37-kube-api-access-76lzx\") pod \"authorino-58cd955675-f9pdh\" (UID: \"6222f410-20d7-4a43-95e7-6d384d673e37\") " pod="kuadrant-system/authorino-58cd955675-f9pdh" Jul 09 19:01:23.183843 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:23.183817 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76lzx\" (UniqueName: \"kubernetes.io/projected/6222f410-20d7-4a43-95e7-6d384d673e37-kube-api-access-76lzx\") pod \"authorino-58cd955675-f9pdh\" (UID: \"6222f410-20d7-4a43-95e7-6d384d673e37\") " pod="kuadrant-system/authorino-58cd955675-f9pdh" Jul 09 19:01:23.276676 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:23.276637 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4k28\" (UniqueName: \"kubernetes.io/projected/7f619dd3-f346-46cd-9e69-91417cfcf9f9-kube-api-access-d4k28\") pod \"authorino-75d556bd77-57gpf\" (UID: \"7f619dd3-f346-46cd-9e69-91417cfcf9f9\") " pod="kuadrant-system/authorino-75d556bd77-57gpf" Jul 09 19:01:23.284560 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:23.284522 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-f9pdh" Jul 09 19:01:23.378090 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:23.378053 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4k28\" (UniqueName: \"kubernetes.io/projected/7f619dd3-f346-46cd-9e69-91417cfcf9f9-kube-api-access-d4k28\") pod \"authorino-75d556bd77-57gpf\" (UID: \"7f619dd3-f346-46cd-9e69-91417cfcf9f9\") " pod="kuadrant-system/authorino-75d556bd77-57gpf" Jul 09 19:01:23.388276 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:23.388247 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4k28\" (UniqueName: \"kubernetes.io/projected/7f619dd3-f346-46cd-9e69-91417cfcf9f9-kube-api-access-d4k28\") pod \"authorino-75d556bd77-57gpf\" (UID: \"7f619dd3-f346-46cd-9e69-91417cfcf9f9\") " pod="kuadrant-system/authorino-75d556bd77-57gpf" Jul 09 19:01:23.420000 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:23.419941 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-57gpf" Jul 09 19:01:23.432612 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:23.432581 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-f9pdh"] Jul 09 19:01:23.435499 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:01:23.435472 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6222f410_20d7_4a43_95e7_6d384d673e37.slice/crio-2897dfa2e658fb44b8c2bc87cd272f0f251dee17cca6ce58c7ff316f56127a82 WatchSource:0}: Error finding container 2897dfa2e658fb44b8c2bc87cd272f0f251dee17cca6ce58c7ff316f56127a82: Status 404 returned error can't find the container with id 2897dfa2e658fb44b8c2bc87cd272f0f251dee17cca6ce58c7ff316f56127a82 Jul 09 19:01:23.577761 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:23.577739 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-57gpf"] Jul 09 19:01:23.579993 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:01:23.579965 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f619dd3_f346_46cd_9e69_91417cfcf9f9.slice/crio-29e152f2d0b9aaab985e7754c3426566626452ad5c6b72c961bc23a98f297329 WatchSource:0}: Error finding container 29e152f2d0b9aaab985e7754c3426566626452ad5c6b72c961bc23a98f297329: Status 404 returned error can't find the container with id 29e152f2d0b9aaab985e7754c3426566626452ad5c6b72c961bc23a98f297329 Jul 09 19:01:23.811009 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:23.810905 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-f9pdh" event={"ID":"6222f410-20d7-4a43-95e7-6d384d673e37","Type":"ContainerStarted","Data":"2897dfa2e658fb44b8c2bc87cd272f0f251dee17cca6ce58c7ff316f56127a82"} Jul 09 19:01:23.811997 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:23.811972 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-57gpf" event={"ID":"7f619dd3-f346-46cd-9e69-91417cfcf9f9","Type":"ContainerStarted","Data":"29e152f2d0b9aaab985e7754c3426566626452ad5c6b72c961bc23a98f297329"} Jul 09 19:01:27.837051 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:27.837007 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-f9pdh" event={"ID":"6222f410-20d7-4a43-95e7-6d384d673e37","Type":"ContainerStarted","Data":"e532f30061e7b66c63430dbe04588939af4ac79a1a49acc23f0eec415a5cc167"} Jul 09 19:01:27.838468 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:27.838447 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-57gpf" event={"ID":"7f619dd3-f346-46cd-9e69-91417cfcf9f9","Type":"ContainerStarted","Data":"f7c3975a0d8241b2d6d5b9e13beb825fd075c945633737f3d4d0db51bd21cca0"} Jul 09 19:01:27.857772 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:27.857727 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-f9pdh" podStartSLOduration=2.300275991 podStartE2EDuration="5.857712983s" totalImagesPullingTime="3.557436992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:01:27.856304468 +0000 UTC m=+472.681467503" watchObservedRunningTime="2026-07-09 19:01:27.857712983 +0000 UTC m=+472.682876020" Jul 09 19:01:27.887744 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:27.887700 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-57gpf" podStartSLOduration=1.487885844 podStartE2EDuration="4.887684166s" totalImagesPullingTime="3.399798322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:01:27.887178735 +0000 UTC m=+472.712341773" watchObservedRunningTime="2026-07-09 19:01:27.887684166 +0000 UTC m=+472.712847202" Jul 09 19:01:27.919674 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:27.919644 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-f9pdh"] Jul 09 19:01:29.848397 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:29.848357 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-f9pdh" podUID="6222f410-20d7-4a43-95e7-6d384d673e37" containerName="authorino" containerID="cri-o://e532f30061e7b66c63430dbe04588939af4ac79a1a49acc23f0eec415a5cc167" gracePeriod=30 Jul 09 19:01:30.093729 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:30.093704 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-f9pdh" Jul 09 19:01:30.138255 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:30.138184 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76lzx\" (UniqueName: \"kubernetes.io/projected/6222f410-20d7-4a43-95e7-6d384d673e37-kube-api-access-76lzx\") pod \"6222f410-20d7-4a43-95e7-6d384d673e37\" (UID: \"6222f410-20d7-4a43-95e7-6d384d673e37\") " Jul 09 19:01:30.140134 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:30.140109 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6222f410-20d7-4a43-95e7-6d384d673e37-kube-api-access-76lzx" (OuterVolumeSpecName: "kube-api-access-76lzx") pod "6222f410-20d7-4a43-95e7-6d384d673e37" (UID: "6222f410-20d7-4a43-95e7-6d384d673e37"). InnerVolumeSpecName "kube-api-access-76lzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:01:30.238603 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:30.238567 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76lzx\" (UniqueName: \"kubernetes.io/projected/6222f410-20d7-4a43-95e7-6d384d673e37-kube-api-access-76lzx\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:01:30.854376 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:30.854338 2569 generic.go:356] "Generic (PLEG): container finished" podID="6222f410-20d7-4a43-95e7-6d384d673e37" containerID="e532f30061e7b66c63430dbe04588939af4ac79a1a49acc23f0eec415a5cc167" exitCode=0 Jul 09 19:01:30.854804 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:30.854381 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-f9pdh" Jul 09 19:01:30.854804 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:30.854415 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-f9pdh" event={"ID":"6222f410-20d7-4a43-95e7-6d384d673e37","Type":"ContainerDied","Data":"e532f30061e7b66c63430dbe04588939af4ac79a1a49acc23f0eec415a5cc167"} Jul 09 19:01:30.854804 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:30.854451 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-f9pdh" event={"ID":"6222f410-20d7-4a43-95e7-6d384d673e37","Type":"ContainerDied","Data":"2897dfa2e658fb44b8c2bc87cd272f0f251dee17cca6ce58c7ff316f56127a82"} Jul 09 19:01:30.854804 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:30.854469 2569 scope.go:117] "RemoveContainer" containerID="e532f30061e7b66c63430dbe04588939af4ac79a1a49acc23f0eec415a5cc167" Jul 09 19:01:30.863864 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:30.863848 2569 scope.go:117] "RemoveContainer" containerID="e532f30061e7b66c63430dbe04588939af4ac79a1a49acc23f0eec415a5cc167" Jul 09 19:01:30.864136 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:01:30.864119 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e532f30061e7b66c63430dbe04588939af4ac79a1a49acc23f0eec415a5cc167\": container with ID starting with e532f30061e7b66c63430dbe04588939af4ac79a1a49acc23f0eec415a5cc167 not found: ID does not exist" containerID="e532f30061e7b66c63430dbe04588939af4ac79a1a49acc23f0eec415a5cc167" Jul 09 19:01:30.864195 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:30.864144 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e532f30061e7b66c63430dbe04588939af4ac79a1a49acc23f0eec415a5cc167"} err="failed to get container status \"e532f30061e7b66c63430dbe04588939af4ac79a1a49acc23f0eec415a5cc167\": rpc error: code = NotFound desc = could not find container \"e532f30061e7b66c63430dbe04588939af4ac79a1a49acc23f0eec415a5cc167\": container with ID starting with e532f30061e7b66c63430dbe04588939af4ac79a1a49acc23f0eec415a5cc167 not found: ID does not exist" Jul 09 19:01:30.876540 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:30.876499 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-f9pdh"] Jul 09 19:01:30.882767 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:30.881050 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-f9pdh"] Jul 09 19:01:31.693433 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:31.693399 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6222f410-20d7-4a43-95e7-6d384d673e37" path="/var/lib/kubelet/pods/6222f410-20d7-4a43-95e7-6d384d673e37/volumes" Jul 09 19:01:52.413071 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.413040 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2rkcx"] Jul 09 19:01:52.413514 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.413356 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6222f410-20d7-4a43-95e7-6d384d673e37" containerName="authorino" Jul 09 19:01:52.413514 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.413368 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6222f410-20d7-4a43-95e7-6d384d673e37" containerName="authorino" Jul 09 19:01:52.413514 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.413430 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6222f410-20d7-4a43-95e7-6d384d673e37" containerName="authorino" Jul 09 19:01:52.418113 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.418096 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2rkcx" Jul 09 19:01:52.427040 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.427016 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2rkcx"] Jul 09 19:01:52.527653 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.527619 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fqrh\" (UniqueName: \"kubernetes.io/projected/6d2c45f1-d81d-411e-8b89-08a4d98e2918-kube-api-access-4fqrh\") pod \"authorino-5d4b67c446-2rkcx\" (UID: \"6d2c45f1-d81d-411e-8b89-08a4d98e2918\") " pod="kuadrant-system/authorino-5d4b67c446-2rkcx" Jul 09 19:01:52.628878 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.628840 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fqrh\" (UniqueName: \"kubernetes.io/projected/6d2c45f1-d81d-411e-8b89-08a4d98e2918-kube-api-access-4fqrh\") pod \"authorino-5d4b67c446-2rkcx\" (UID: \"6d2c45f1-d81d-411e-8b89-08a4d98e2918\") " pod="kuadrant-system/authorino-5d4b67c446-2rkcx" Jul 09 19:01:52.643162 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.643135 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fqrh\" (UniqueName: \"kubernetes.io/projected/6d2c45f1-d81d-411e-8b89-08a4d98e2918-kube-api-access-4fqrh\") pod \"authorino-5d4b67c446-2rkcx\" (UID: \"6d2c45f1-d81d-411e-8b89-08a4d98e2918\") " pod="kuadrant-system/authorino-5d4b67c446-2rkcx" Jul 09 19:01:52.696163 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.696084 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2rkcx"] Jul 09 19:01:52.696342 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.696330 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2rkcx" Jul 09 19:01:52.732539 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.732509 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56db8d7945-s9np7"] Jul 09 19:01:52.739163 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.737504 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56db8d7945-s9np7" Jul 09 19:01:52.741606 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.741421 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 09 19:01:52.743553 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.743530 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56db8d7945-s9np7"] Jul 09 19:01:52.830538 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.830498 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dad8aa30-3cc0-4626-bc03-155d60236d38-tls-cert\") pod \"authorino-56db8d7945-s9np7\" (UID: \"dad8aa30-3cc0-4626-bc03-155d60236d38\") " pod="kuadrant-system/authorino-56db8d7945-s9np7" Jul 09 19:01:52.830693 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.830567 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ww6n\" (UniqueName: \"kubernetes.io/projected/dad8aa30-3cc0-4626-bc03-155d60236d38-kube-api-access-8ww6n\") pod \"authorino-56db8d7945-s9np7\" (UID: \"dad8aa30-3cc0-4626-bc03-155d60236d38\") " pod="kuadrant-system/authorino-56db8d7945-s9np7" Jul 09 19:01:52.834304 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.834283 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2rkcx"] Jul 09 19:01:52.836149 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:01:52.836120 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d2c45f1_d81d_411e_8b89_08a4d98e2918.slice/crio-441555ed321b174f771dc12e135c3064d451911ca5802c0e452388db8896f4c7 WatchSource:0}: Error finding container 441555ed321b174f771dc12e135c3064d451911ca5802c0e452388db8896f4c7: Status 404 returned error can't find the container with id 441555ed321b174f771dc12e135c3064d451911ca5802c0e452388db8896f4c7 Jul 09 19:01:52.931367 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.931334 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ww6n\" (UniqueName: \"kubernetes.io/projected/dad8aa30-3cc0-4626-bc03-155d60236d38-kube-api-access-8ww6n\") pod \"authorino-56db8d7945-s9np7\" (UID: \"dad8aa30-3cc0-4626-bc03-155d60236d38\") " pod="kuadrant-system/authorino-56db8d7945-s9np7" Jul 09 19:01:52.931543 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.931402 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dad8aa30-3cc0-4626-bc03-155d60236d38-tls-cert\") pod \"authorino-56db8d7945-s9np7\" (UID: \"dad8aa30-3cc0-4626-bc03-155d60236d38\") " pod="kuadrant-system/authorino-56db8d7945-s9np7" Jul 09 19:01:52.933731 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.933702 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dad8aa30-3cc0-4626-bc03-155d60236d38-tls-cert\") pod \"authorino-56db8d7945-s9np7\" (UID: \"dad8aa30-3cc0-4626-bc03-155d60236d38\") " pod="kuadrant-system/authorino-56db8d7945-s9np7" Jul 09 19:01:52.943760 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.943732 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ww6n\" (UniqueName: \"kubernetes.io/projected/dad8aa30-3cc0-4626-bc03-155d60236d38-kube-api-access-8ww6n\") pod \"authorino-56db8d7945-s9np7\" (UID: \"dad8aa30-3cc0-4626-bc03-155d60236d38\") " pod="kuadrant-system/authorino-56db8d7945-s9np7" Jul 09 19:01:52.963285 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:52.963257 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2rkcx" event={"ID":"6d2c45f1-d81d-411e-8b89-08a4d98e2918","Type":"ContainerStarted","Data":"441555ed321b174f771dc12e135c3064d451911ca5802c0e452388db8896f4c7"} Jul 09 19:01:53.051746 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:53.051695 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56db8d7945-s9np7" Jul 09 19:01:53.180868 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:53.180844 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56db8d7945-s9np7"] Jul 09 19:01:53.184109 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:01:53.184083 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddad8aa30_3cc0_4626_bc03_155d60236d38.slice/crio-9046b11a1ad336792b81f3015ead193feecab748732fa4ac4429c109f932c211 WatchSource:0}: Error finding container 9046b11a1ad336792b81f3015ead193feecab748732fa4ac4429c109f932c211: Status 404 returned error can't find the container with id 9046b11a1ad336792b81f3015ead193feecab748732fa4ac4429c109f932c211 Jul 09 19:01:53.970443 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:53.970410 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2rkcx" event={"ID":"6d2c45f1-d81d-411e-8b89-08a4d98e2918","Type":"ContainerStarted","Data":"e039426aff6369997338105c9f0a7422754dd494b5cf3cd298c11fb503f28275"} Jul 09 19:01:53.970844 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:53.970458 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-2rkcx" podUID="6d2c45f1-d81d-411e-8b89-08a4d98e2918" containerName="authorino" containerID="cri-o://e039426aff6369997338105c9f0a7422754dd494b5cf3cd298c11fb503f28275" gracePeriod=30 Jul 09 19:01:53.972056 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:53.972035 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56db8d7945-s9np7" event={"ID":"dad8aa30-3cc0-4626-bc03-155d60236d38","Type":"ContainerStarted","Data":"9c374d89475cb6e7d421e92ce6ac6e0845827aa69766395d7b52663a3e207bc8"} Jul 09 19:01:53.972193 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:53.972061 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56db8d7945-s9np7" event={"ID":"dad8aa30-3cc0-4626-bc03-155d60236d38","Type":"ContainerStarted","Data":"9046b11a1ad336792b81f3015ead193feecab748732fa4ac4429c109f932c211"} Jul 09 19:01:53.993482 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:53.993441 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-2rkcx" podStartSLOduration=1.657422561 podStartE2EDuration="1.993426976s" totalImagesPullingTime="336.004415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:01:53.991238515 +0000 UTC m=+498.816401549" watchObservedRunningTime="2026-07-09 19:01:53.993426976 +0000 UTC m=+498.818590048" Jul 09 19:01:54.009484 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.009431 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-56db8d7945-s9np7" podStartSLOduration=1.524335269 podStartE2EDuration="2.009414851s" totalImagesPullingTime="485.079582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:01:54.007641077 +0000 UTC m=+498.832804115" watchObservedRunningTime="2026-07-09 19:01:54.009414851 +0000 UTC m=+498.834577888" Jul 09 19:01:54.040870 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.040841 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-57gpf"] Jul 09 19:01:54.041073 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.041055 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-57gpf" podUID="7f619dd3-f346-46cd-9e69-91417cfcf9f9" containerName="authorino" containerID="cri-o://f7c3975a0d8241b2d6d5b9e13beb825fd075c945633737f3d4d0db51bd21cca0" gracePeriod=30 Jul 09 19:01:54.287589 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.287561 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2rkcx" Jul 09 19:01:54.295284 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.295263 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-57gpf" Jul 09 19:01:54.343098 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.343062 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fqrh\" (UniqueName: \"kubernetes.io/projected/6d2c45f1-d81d-411e-8b89-08a4d98e2918-kube-api-access-4fqrh\") pod \"6d2c45f1-d81d-411e-8b89-08a4d98e2918\" (UID: \"6d2c45f1-d81d-411e-8b89-08a4d98e2918\") " Jul 09 19:01:54.345040 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.345016 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2c45f1-d81d-411e-8b89-08a4d98e2918-kube-api-access-4fqrh" (OuterVolumeSpecName: "kube-api-access-4fqrh") pod "6d2c45f1-d81d-411e-8b89-08a4d98e2918" (UID: "6d2c45f1-d81d-411e-8b89-08a4d98e2918"). InnerVolumeSpecName "kube-api-access-4fqrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:01:54.443539 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.443500 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4k28\" (UniqueName: \"kubernetes.io/projected/7f619dd3-f346-46cd-9e69-91417cfcf9f9-kube-api-access-d4k28\") pod \"7f619dd3-f346-46cd-9e69-91417cfcf9f9\" (UID: \"7f619dd3-f346-46cd-9e69-91417cfcf9f9\") " Jul 09 19:01:54.443703 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.443696 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fqrh\" (UniqueName: \"kubernetes.io/projected/6d2c45f1-d81d-411e-8b89-08a4d98e2918-kube-api-access-4fqrh\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:01:54.445490 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.445468 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f619dd3-f346-46cd-9e69-91417cfcf9f9-kube-api-access-d4k28" (OuterVolumeSpecName: "kube-api-access-d4k28") pod "7f619dd3-f346-46cd-9e69-91417cfcf9f9" (UID: "7f619dd3-f346-46cd-9e69-91417cfcf9f9"). InnerVolumeSpecName "kube-api-access-d4k28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:01:54.544416 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.544327 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4k28\" (UniqueName: \"kubernetes.io/projected/7f619dd3-f346-46cd-9e69-91417cfcf9f9-kube-api-access-d4k28\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:01:54.980347 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.980312 2569 generic.go:356] "Generic (PLEG): container finished" podID="7f619dd3-f346-46cd-9e69-91417cfcf9f9" containerID="f7c3975a0d8241b2d6d5b9e13beb825fd075c945633737f3d4d0db51bd21cca0" exitCode=0 Jul 09 19:01:54.980787 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.980353 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-57gpf" Jul 09 19:01:54.980787 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.980393 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-57gpf" event={"ID":"7f619dd3-f346-46cd-9e69-91417cfcf9f9","Type":"ContainerDied","Data":"f7c3975a0d8241b2d6d5b9e13beb825fd075c945633737f3d4d0db51bd21cca0"} Jul 09 19:01:54.980787 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.980431 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-57gpf" event={"ID":"7f619dd3-f346-46cd-9e69-91417cfcf9f9","Type":"ContainerDied","Data":"29e152f2d0b9aaab985e7754c3426566626452ad5c6b72c961bc23a98f297329"} Jul 09 19:01:54.980787 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.980456 2569 scope.go:117] "RemoveContainer" containerID="f7c3975a0d8241b2d6d5b9e13beb825fd075c945633737f3d4d0db51bd21cca0" Jul 09 19:01:54.981770 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.981603 2569 generic.go:356] "Generic (PLEG): container finished" podID="6d2c45f1-d81d-411e-8b89-08a4d98e2918" containerID="e039426aff6369997338105c9f0a7422754dd494b5cf3cd298c11fb503f28275" exitCode=0 Jul 09 19:01:54.981770 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.981638 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2rkcx" Jul 09 19:01:54.981770 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.981665 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2rkcx" event={"ID":"6d2c45f1-d81d-411e-8b89-08a4d98e2918","Type":"ContainerDied","Data":"e039426aff6369997338105c9f0a7422754dd494b5cf3cd298c11fb503f28275"} Jul 09 19:01:54.981770 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.981693 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2rkcx" event={"ID":"6d2c45f1-d81d-411e-8b89-08a4d98e2918","Type":"ContainerDied","Data":"441555ed321b174f771dc12e135c3064d451911ca5802c0e452388db8896f4c7"} Jul 09 19:01:54.989979 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.989960 2569 scope.go:117] "RemoveContainer" containerID="f7c3975a0d8241b2d6d5b9e13beb825fd075c945633737f3d4d0db51bd21cca0" Jul 09 19:01:54.990258 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:01:54.990237 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7c3975a0d8241b2d6d5b9e13beb825fd075c945633737f3d4d0db51bd21cca0\": container with ID starting with f7c3975a0d8241b2d6d5b9e13beb825fd075c945633737f3d4d0db51bd21cca0 not found: ID does not exist" containerID="f7c3975a0d8241b2d6d5b9e13beb825fd075c945633737f3d4d0db51bd21cca0" Jul 09 19:01:54.990307 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.990266 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7c3975a0d8241b2d6d5b9e13beb825fd075c945633737f3d4d0db51bd21cca0"} err="failed to get container status \"f7c3975a0d8241b2d6d5b9e13beb825fd075c945633737f3d4d0db51bd21cca0\": rpc error: code = NotFound desc = could not find container \"f7c3975a0d8241b2d6d5b9e13beb825fd075c945633737f3d4d0db51bd21cca0\": container with ID starting with f7c3975a0d8241b2d6d5b9e13beb825fd075c945633737f3d4d0db51bd21cca0 not found: ID does not exist" Jul 09 19:01:54.990307 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.990281 2569 scope.go:117] "RemoveContainer" containerID="e039426aff6369997338105c9f0a7422754dd494b5cf3cd298c11fb503f28275" Jul 09 19:01:54.997807 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.997788 2569 scope.go:117] "RemoveContainer" containerID="e039426aff6369997338105c9f0a7422754dd494b5cf3cd298c11fb503f28275" Jul 09 19:01:54.998127 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:01:54.998110 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e039426aff6369997338105c9f0a7422754dd494b5cf3cd298c11fb503f28275\": container with ID starting with e039426aff6369997338105c9f0a7422754dd494b5cf3cd298c11fb503f28275 not found: ID does not exist" containerID="e039426aff6369997338105c9f0a7422754dd494b5cf3cd298c11fb503f28275" Jul 09 19:01:54.998198 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:54.998136 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e039426aff6369997338105c9f0a7422754dd494b5cf3cd298c11fb503f28275"} err="failed to get container status \"e039426aff6369997338105c9f0a7422754dd494b5cf3cd298c11fb503f28275\": rpc error: code = NotFound desc = could not find container \"e039426aff6369997338105c9f0a7422754dd494b5cf3cd298c11fb503f28275\": container with ID starting with e039426aff6369997338105c9f0a7422754dd494b5cf3cd298c11fb503f28275 not found: ID does not exist" Jul 09 19:01:55.004449 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:55.004418 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2rkcx"] Jul 09 19:01:55.008172 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:55.008152 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2rkcx"] Jul 09 19:01:55.018833 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:55.018812 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-57gpf"] Jul 09 19:01:55.022474 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:55.022452 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-57gpf"] Jul 09 19:01:55.698731 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:55.698698 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d2c45f1-d81d-411e-8b89-08a4d98e2918" path="/var/lib/kubelet/pods/6d2c45f1-d81d-411e-8b89-08a4d98e2918/volumes" Jul 09 19:01:55.699045 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:01:55.699032 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f619dd3-f346-46cd-9e69-91417cfcf9f9" path="/var/lib/kubelet/pods/7f619dd3-f346-46cd-9e69-91417cfcf9f9/volumes" Jul 09 19:02:37.094319 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.094285 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-nxrmk"] Jul 09 19:02:37.094694 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.094603 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f619dd3-f346-46cd-9e69-91417cfcf9f9" containerName="authorino" Jul 09 19:02:37.094694 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.094614 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f619dd3-f346-46cd-9e69-91417cfcf9f9" containerName="authorino" Jul 09 19:02:37.094694 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.094623 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2c45f1-d81d-411e-8b89-08a4d98e2918" containerName="authorino" Jul 09 19:02:37.094694 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.094628 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2c45f1-d81d-411e-8b89-08a4d98e2918" containerName="authorino" Jul 09 19:02:37.094694 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.094675 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f619dd3-f346-46cd-9e69-91417cfcf9f9" containerName="authorino" Jul 09 19:02:37.094694 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.094681 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d2c45f1-d81d-411e-8b89-08a4d98e2918" containerName="authorino" Jul 09 19:02:37.097406 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.097388 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-nxrmk" Jul 09 19:02:37.099937 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.099917 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 09 19:02:37.100053 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.099996 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rmfgg\"" Jul 09 19:02:37.106071 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.106048 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-nxrmk"] Jul 09 19:02:37.190803 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.190767 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52-plugins-config-volume\") pod \"payload-pre-processing-7964457696-nxrmk\" (UID: \"c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52\") " pod="openshift-ingress/payload-pre-processing-7964457696-nxrmk" Jul 09 19:02:37.191001 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.190810 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn8xb\" (UniqueName: \"kubernetes.io/projected/c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52-kube-api-access-kn8xb\") pod \"payload-pre-processing-7964457696-nxrmk\" (UID: \"c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52\") " pod="openshift-ingress/payload-pre-processing-7964457696-nxrmk" Jul 09 19:02:37.291337 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.291298 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52-plugins-config-volume\") pod \"payload-pre-processing-7964457696-nxrmk\" (UID: \"c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52\") " pod="openshift-ingress/payload-pre-processing-7964457696-nxrmk" Jul 09 19:02:37.291537 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.291396 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn8xb\" (UniqueName: \"kubernetes.io/projected/c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52-kube-api-access-kn8xb\") pod \"payload-pre-processing-7964457696-nxrmk\" (UID: \"c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52\") " pod="openshift-ingress/payload-pre-processing-7964457696-nxrmk" Jul 09 19:02:37.292029 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.292011 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52-plugins-config-volume\") pod \"payload-pre-processing-7964457696-nxrmk\" (UID: \"c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52\") " pod="openshift-ingress/payload-pre-processing-7964457696-nxrmk" Jul 09 19:02:37.346666 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.346592 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn8xb\" (UniqueName: \"kubernetes.io/projected/c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52-kube-api-access-kn8xb\") pod \"payload-pre-processing-7964457696-nxrmk\" (UID: \"c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52\") " pod="openshift-ingress/payload-pre-processing-7964457696-nxrmk" Jul 09 19:02:37.408712 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.408682 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-nxrmk" Jul 09 19:02:37.533480 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:37.533457 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-nxrmk"] Jul 09 19:02:37.535240 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:02:37.535210 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ab4d25_34fa_4a94_9c0d_297e8b8f8b52.slice/crio-4925ae52cf039422fb3674b4f6f0608ef41a7d5c417e43042cc01eded07c05ac WatchSource:0}: Error finding container 4925ae52cf039422fb3674b4f6f0608ef41a7d5c417e43042cc01eded07c05ac: Status 404 returned error can't find the container with id 4925ae52cf039422fb3674b4f6f0608ef41a7d5c417e43042cc01eded07c05ac Jul 09 19:02:38.185158 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:38.185122 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-nxrmk" event={"ID":"c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52","Type":"ContainerStarted","Data":"4925ae52cf039422fb3674b4f6f0608ef41a7d5c417e43042cc01eded07c05ac"} Jul 09 19:02:39.191721 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:39.191630 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-nxrmk" event={"ID":"c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52","Type":"ContainerStarted","Data":"b0b5f5fdea2933f81b230e8ed4d7d9fd107ca87cd1a50a5ea91694b702f17638"} Jul 09 19:02:39.192082 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:39.191736 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-nxrmk" Jul 09 19:02:39.210218 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:39.210170 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-nxrmk" podStartSLOduration=0.810099513 podStartE2EDuration="2.210154649s" totalImagesPullingTime="1.400055136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:02:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:02:39.207694371 +0000 UTC m=+544.032857409" watchObservedRunningTime="2026-07-09 19:02:39.210154649 +0000 UTC m=+544.035317683" Jul 09 19:02:50.202497 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:02:50.202462 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-nxrmk" Jul 09 19:03:04.353244 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.353201 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc"] Jul 09 19:03:04.355966 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.355929 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.359506 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.359481 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 19:03:04.359639 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.359480 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 19:03:04.359639 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.359573 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l6hgx\"" Jul 09 19:03:04.359639 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.359517 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 09 19:03:04.368098 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.368073 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc"] Jul 09 19:03:04.523283 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.523246 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.523283 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.523289 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.523490 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.523314 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.523490 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.523444 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.523490 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.523483 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.523589 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.523520 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgnhb\" (UniqueName: \"kubernetes.io/projected/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-kube-api-access-pgnhb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.624228 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.624137 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.624228 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.624184 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.624228 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.624218 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgnhb\" (UniqueName: \"kubernetes.io/projected/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-kube-api-access-pgnhb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.624483 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.624269 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.624483 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.624294 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.624483 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.624319 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.624626 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.624577 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.624626 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.624589 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.624737 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.624668 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.626513 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.626483 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.626699 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.626683 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.632942 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.632922 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgnhb\" (UniqueName: \"kubernetes.io/projected/f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712-kube-api-access-pgnhb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc\" (UID: \"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.666984 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.666933 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:04.795754 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:04.795726 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc"] Jul 09 19:03:04.797815 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:03:04.797786 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ea3f9d_42a8_4a41_8c01_9b34d5d2a712.slice/crio-dbe8a40740cdb8dbe3e5847c16a7babc54ad42d3c2804e59aacc3fdd132ca3ca WatchSource:0}: Error finding container dbe8a40740cdb8dbe3e5847c16a7babc54ad42d3c2804e59aacc3fdd132ca3ca: Status 404 returned error can't find the container with id dbe8a40740cdb8dbe3e5847c16a7babc54ad42d3c2804e59aacc3fdd132ca3ca Jul 09 19:03:05.319801 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:05.319734 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" event={"ID":"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712","Type":"ContainerStarted","Data":"dbe8a40740cdb8dbe3e5847c16a7babc54ad42d3c2804e59aacc3fdd132ca3ca"} Jul 09 19:03:06.326430 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:06.326335 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" event={"ID":"f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712","Type":"ContainerStarted","Data":"ec2936104c8f238498352db03c0ef61fb862994d8799dcb5e2bf70b79d891949"} Jul 09 19:03:06.342905 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:06.342851 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" podStartSLOduration=1.145971144 podStartE2EDuration="2.342832485s" totalImagesPullingTime="1.196861341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:03:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:06.342672252 +0000 UTC m=+571.167835312" watchObservedRunningTime="2026-07-09 19:03:06.342832485 +0000 UTC m=+571.167995524" Jul 09 19:03:14.668049 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:14.667994 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:14.668049 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:14.668066 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:14.680561 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:14.680530 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:15.380063 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:15.380037 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc" Jul 09 19:03:35.607193 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:35.607159 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 19:03:35.610147 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:35.610125 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 19:03:37.350122 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.350091 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5"] Jul 09 19:03:37.352300 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.352284 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.354886 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.354862 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 09 19:03:37.365636 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.365612 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5"] Jul 09 19:03:37.397499 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.397471 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/503b4318-56c2-4696-8b5f-203b229ba825-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.397632 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.397511 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svgm5\" (UniqueName: \"kubernetes.io/projected/503b4318-56c2-4696-8b5f-203b229ba825-kube-api-access-svgm5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.397632 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.397529 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/503b4318-56c2-4696-8b5f-203b229ba825-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.397632 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.397607 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/503b4318-56c2-4696-8b5f-203b229ba825-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.397632 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.397624 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/503b4318-56c2-4696-8b5f-203b229ba825-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.397766 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.397689 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/503b4318-56c2-4696-8b5f-203b229ba825-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.499071 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.499039 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/503b4318-56c2-4696-8b5f-203b229ba825-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.499249 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.499078 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svgm5\" (UniqueName: \"kubernetes.io/projected/503b4318-56c2-4696-8b5f-203b229ba825-kube-api-access-svgm5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.499249 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.499098 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/503b4318-56c2-4696-8b5f-203b229ba825-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.499249 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.499140 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/503b4318-56c2-4696-8b5f-203b229ba825-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.499249 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.499154 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/503b4318-56c2-4696-8b5f-203b229ba825-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.499249 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.499181 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/503b4318-56c2-4696-8b5f-203b229ba825-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.499557 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.499532 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/503b4318-56c2-4696-8b5f-203b229ba825-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.499617 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.499563 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/503b4318-56c2-4696-8b5f-203b229ba825-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.499617 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.499574 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/503b4318-56c2-4696-8b5f-203b229ba825-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.501380 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.501361 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/503b4318-56c2-4696-8b5f-203b229ba825-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.501649 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.501630 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/503b4318-56c2-4696-8b5f-203b229ba825-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.507065 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.507042 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svgm5\" (UniqueName: \"kubernetes.io/projected/503b4318-56c2-4696-8b5f-203b229ba825-kube-api-access-svgm5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b5cd5\" (UID: \"503b4318-56c2-4696-8b5f-203b229ba825\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.663591 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.663510 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:37.792847 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:37.792823 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5"] Jul 09 19:03:37.794696 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:03:37.794665 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503b4318_56c2_4696_8b5f_203b229ba825.slice/crio-8e7b80241c0ad3259a2d822d5ec7aa39c3ea897d9b76fc3b729209e490a37ebc WatchSource:0}: Error finding container 8e7b80241c0ad3259a2d822d5ec7aa39c3ea897d9b76fc3b729209e490a37ebc: Status 404 returned error can't find the container with id 8e7b80241c0ad3259a2d822d5ec7aa39c3ea897d9b76fc3b729209e490a37ebc Jul 09 19:03:38.489514 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:38.489471 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" event={"ID":"503b4318-56c2-4696-8b5f-203b229ba825","Type":"ContainerStarted","Data":"bb3491205589fde4f87218ebc943eaf672b78fdbc56d344472cc46b145670504"} Jul 09 19:03:38.489984 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:38.489572 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" event={"ID":"503b4318-56c2-4696-8b5f-203b229ba825","Type":"ContainerStarted","Data":"8e7b80241c0ad3259a2d822d5ec7aa39c3ea897d9b76fc3b729209e490a37ebc"} Jul 09 19:03:38.515036 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:38.514974 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" podStartSLOduration=1.280198304 podStartE2EDuration="1.514934569s" totalImagesPullingTime="234.736265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:38.510569406 +0000 UTC m=+603.335732443" watchObservedRunningTime="2026-07-09 19:03:38.514934569 +0000 UTC m=+603.340097606" Jul 09 19:03:47.663811 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:47.663770 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:47.664222 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:47.663842 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:47.676407 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:47.676382 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:48.550828 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:48.550802 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b5cd5" Jul 09 19:03:55.847807 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:55.847772 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q"] Jul 09 19:03:55.855776 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:55.855753 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:55.859967 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:55.859920 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 09 19:03:55.862579 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:55.862553 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q"] Jul 09 19:03:55.961702 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:55.961667 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtnwj\" (UniqueName: \"kubernetes.io/projected/c602f2e4-e262-4830-b826-58070d1e5caf-kube-api-access-rtnwj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:55.961702 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:55.961714 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c602f2e4-e262-4830-b826-58070d1e5caf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:55.961930 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:55.961760 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c602f2e4-e262-4830-b826-58070d1e5caf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:55.961930 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:55.961802 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c602f2e4-e262-4830-b826-58070d1e5caf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:55.961930 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:55.961847 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c602f2e4-e262-4830-b826-58070d1e5caf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:55.961930 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:55.961884 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c602f2e4-e262-4830-b826-58070d1e5caf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:56.062532 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.062494 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c602f2e4-e262-4830-b826-58070d1e5caf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:56.062722 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.062548 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtnwj\" (UniqueName: \"kubernetes.io/projected/c602f2e4-e262-4830-b826-58070d1e5caf-kube-api-access-rtnwj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:56.062722 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.062574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c602f2e4-e262-4830-b826-58070d1e5caf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:56.062722 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.062590 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c602f2e4-e262-4830-b826-58070d1e5caf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:56.062722 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.062609 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c602f2e4-e262-4830-b826-58070d1e5caf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:56.062940 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.062768 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c602f2e4-e262-4830-b826-58070d1e5caf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:56.065578 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.063448 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c602f2e4-e262-4830-b826-58070d1e5caf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:56.065578 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.063462 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c602f2e4-e262-4830-b826-58070d1e5caf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:56.065578 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.063562 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c602f2e4-e262-4830-b826-58070d1e5caf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:56.069902 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.069773 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c602f2e4-e262-4830-b826-58070d1e5caf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:56.071001 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.070976 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c602f2e4-e262-4830-b826-58070d1e5caf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:56.071152 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.071132 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtnwj\" (UniqueName: \"kubernetes.io/projected/c602f2e4-e262-4830-b826-58070d1e5caf-kube-api-access-rtnwj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h476q\" (UID: \"c602f2e4-e262-4830-b826-58070d1e5caf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:56.170866 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.170764 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:03:56.303137 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.303109 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q"] Jul 09 19:03:56.304738 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:03:56.304712 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc602f2e4_e262_4830_b826_58070d1e5caf.slice/crio-b7feb1846c87caf83b2c4f03ba14619112ec79cf702989f8646d907732ca9085 WatchSource:0}: Error finding container b7feb1846c87caf83b2c4f03ba14619112ec79cf702989f8646d907732ca9085: Status 404 returned error can't find the container with id b7feb1846c87caf83b2c4f03ba14619112ec79cf702989f8646d907732ca9085 Jul 09 19:03:56.306040 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.306013 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 19:03:56.580624 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:56.580596 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" event={"ID":"c602f2e4-e262-4830-b826-58070d1e5caf","Type":"ContainerStarted","Data":"b7feb1846c87caf83b2c4f03ba14619112ec79cf702989f8646d907732ca9085"} Jul 09 19:03:57.588621 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:57.588547 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" event={"ID":"c602f2e4-e262-4830-b826-58070d1e5caf","Type":"ContainerStarted","Data":"fcb0a10cdd37d6bb2cfc05583156639848bda3a32e334d7f4f37f0370188e27a"} Jul 09 19:03:57.606436 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:03:57.606386 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" podStartSLOduration=2.3777589470000002 podStartE2EDuration="2.60637062s" totalImagesPullingTime="228.611673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:57.605055226 +0000 UTC m=+622.430218264" watchObservedRunningTime="2026-07-09 19:03:57.60637062 +0000 UTC m=+622.431533657" Jul 09 19:04:06.171675 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:06.171623 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:04:06.172167 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:06.171696 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:04:06.184163 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:06.184136 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:04:06.642773 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:06.642747 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h476q" Jul 09 19:04:12.035024 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.034992 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h"] Jul 09 19:04:12.037564 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.037546 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.040081 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.040063 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 09 19:04:12.049389 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.049365 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h"] Jul 09 19:04:12.102072 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.102042 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be36501-e4c2-4d76-9b42-fa7578ab7932-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.102224 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.102102 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be36501-e4c2-4d76-9b42-fa7578ab7932-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.102224 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.102119 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be36501-e4c2-4d76-9b42-fa7578ab7932-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.102224 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.102156 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be36501-e4c2-4d76-9b42-fa7578ab7932-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.102224 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.102180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf88c\" (UniqueName: \"kubernetes.io/projected/3be36501-e4c2-4d76-9b42-fa7578ab7932-kube-api-access-wf88c\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.102224 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.102214 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be36501-e4c2-4d76-9b42-fa7578ab7932-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.202556 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.202518 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf88c\" (UniqueName: \"kubernetes.io/projected/3be36501-e4c2-4d76-9b42-fa7578ab7932-kube-api-access-wf88c\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.202736 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.202569 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be36501-e4c2-4d76-9b42-fa7578ab7932-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.202736 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.202607 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be36501-e4c2-4d76-9b42-fa7578ab7932-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.202736 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.202667 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be36501-e4c2-4d76-9b42-fa7578ab7932-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.202736 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.202691 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be36501-e4c2-4d76-9b42-fa7578ab7932-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.202736 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.202722 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be36501-e4c2-4d76-9b42-fa7578ab7932-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.203150 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.203130 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be36501-e4c2-4d76-9b42-fa7578ab7932-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.203218 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.203149 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be36501-e4c2-4d76-9b42-fa7578ab7932-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.203218 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.203201 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be36501-e4c2-4d76-9b42-fa7578ab7932-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.204906 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.204887 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be36501-e4c2-4d76-9b42-fa7578ab7932-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.205206 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.205189 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be36501-e4c2-4d76-9b42-fa7578ab7932-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.209857 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.209833 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf88c\" (UniqueName: \"kubernetes.io/projected/3be36501-e4c2-4d76-9b42-fa7578ab7932-kube-api-access-wf88c\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h\" (UID: \"3be36501-e4c2-4d76-9b42-fa7578ab7932\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.347181 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.347097 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:12.479314 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.479289 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h"] Jul 09 19:04:12.481278 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:04:12.481242 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3be36501_e4c2_4d76_9b42_fa7578ab7932.slice/crio-338a58e4d9677068e46516e416bcd13e915709850b378449982ca792f4a70dcf WatchSource:0}: Error finding container 338a58e4d9677068e46516e416bcd13e915709850b378449982ca792f4a70dcf: Status 404 returned error can't find the container with id 338a58e4d9677068e46516e416bcd13e915709850b378449982ca792f4a70dcf Jul 09 19:04:12.664319 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:12.664287 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" event={"ID":"3be36501-e4c2-4d76-9b42-fa7578ab7932","Type":"ContainerStarted","Data":"338a58e4d9677068e46516e416bcd13e915709850b378449982ca792f4a70dcf"} Jul 09 19:04:13.671638 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:13.671600 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" event={"ID":"3be36501-e4c2-4d76-9b42-fa7578ab7932","Type":"ContainerStarted","Data":"9a75f301c707467fa1fa60df5bfb662af510d4ca9fd67ce1888d6e50dc520623"} Jul 09 19:04:13.694677 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:13.694626 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" podStartSLOduration=1.5242322270000002 podStartE2EDuration="1.694610826s" totalImagesPullingTime="170.378599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:04:13.692833138 +0000 UTC m=+638.517996176" watchObservedRunningTime="2026-07-09 19:04:13.694610826 +0000 UTC m=+638.519773863" Jul 09 19:04:22.347429 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:22.347382 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:22.347853 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:22.347452 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:22.360033 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:22.360007 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:22.733031 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:22.733003 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h" Jul 09 19:04:42.948443 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:42.948409 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-867fd7fc68-r5r5s"] Jul 09 19:04:42.950989 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:42.950971 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-867fd7fc68-r5r5s" Jul 09 19:04:42.960165 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:42.960144 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-867fd7fc68-r5r5s"] Jul 09 19:04:43.075824 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:43.075786 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22qzp\" (UniqueName: \"kubernetes.io/projected/dcab9734-a390-4768-8dc6-13ed97aa6b8d-kube-api-access-22qzp\") pod \"authorino-867fd7fc68-r5r5s\" (UID: \"dcab9734-a390-4768-8dc6-13ed97aa6b8d\") " pod="kuadrant-system/authorino-867fd7fc68-r5r5s" Jul 09 19:04:43.076101 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:43.075872 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dcab9734-a390-4768-8dc6-13ed97aa6b8d-tls-cert\") pod \"authorino-867fd7fc68-r5r5s\" (UID: \"dcab9734-a390-4768-8dc6-13ed97aa6b8d\") " pod="kuadrant-system/authorino-867fd7fc68-r5r5s" Jul 09 19:04:43.177127 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:43.177091 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22qzp\" (UniqueName: \"kubernetes.io/projected/dcab9734-a390-4768-8dc6-13ed97aa6b8d-kube-api-access-22qzp\") pod \"authorino-867fd7fc68-r5r5s\" (UID: \"dcab9734-a390-4768-8dc6-13ed97aa6b8d\") " pod="kuadrant-system/authorino-867fd7fc68-r5r5s" Jul 09 19:04:43.177306 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:43.177177 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dcab9734-a390-4768-8dc6-13ed97aa6b8d-tls-cert\") pod \"authorino-867fd7fc68-r5r5s\" (UID: \"dcab9734-a390-4768-8dc6-13ed97aa6b8d\") " pod="kuadrant-system/authorino-867fd7fc68-r5r5s" Jul 09 19:04:43.179602 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:43.179571 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dcab9734-a390-4768-8dc6-13ed97aa6b8d-tls-cert\") pod \"authorino-867fd7fc68-r5r5s\" (UID: \"dcab9734-a390-4768-8dc6-13ed97aa6b8d\") " pod="kuadrant-system/authorino-867fd7fc68-r5r5s" Jul 09 19:04:43.184565 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:43.184542 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22qzp\" (UniqueName: \"kubernetes.io/projected/dcab9734-a390-4768-8dc6-13ed97aa6b8d-kube-api-access-22qzp\") pod \"authorino-867fd7fc68-r5r5s\" (UID: \"dcab9734-a390-4768-8dc6-13ed97aa6b8d\") " pod="kuadrant-system/authorino-867fd7fc68-r5r5s" Jul 09 19:04:43.260718 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:43.260688 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-867fd7fc68-r5r5s" Jul 09 19:04:43.393180 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:43.393156 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-867fd7fc68-r5r5s"] Jul 09 19:04:43.394933 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:04:43.394903 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcab9734_a390_4768_8dc6_13ed97aa6b8d.slice/crio-e8a587297e121739b779b4a8be1b92cf10a9a5a43d46ce61f1b8bedf479e403a WatchSource:0}: Error finding container e8a587297e121739b779b4a8be1b92cf10a9a5a43d46ce61f1b8bedf479e403a: Status 404 returned error can't find the container with id e8a587297e121739b779b4a8be1b92cf10a9a5a43d46ce61f1b8bedf479e403a Jul 09 19:04:43.830872 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:43.830840 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-867fd7fc68-r5r5s" event={"ID":"dcab9734-a390-4768-8dc6-13ed97aa6b8d","Type":"ContainerStarted","Data":"e8a587297e121739b779b4a8be1b92cf10a9a5a43d46ce61f1b8bedf479e403a"} Jul 09 19:04:44.839439 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:44.839397 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-867fd7fc68-r5r5s" event={"ID":"dcab9734-a390-4768-8dc6-13ed97aa6b8d","Type":"ContainerStarted","Data":"f1168b826b604957362e123f7140fb1458e6198acbe16776468bbf944dbb8bce"} Jul 09 19:04:44.899727 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:44.899657 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-867fd7fc68-r5r5s" podStartSLOduration=2.30523375 podStartE2EDuration="2.8996339s" totalImagesPullingTime="594.40015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:04:44.890204802 +0000 UTC m=+669.715367841" watchObservedRunningTime="2026-07-09 19:04:44.8996339 +0000 UTC m=+669.724796937" Jul 09 19:04:44.978975 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:44.978927 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56db8d7945-s9np7"] Jul 09 19:04:44.981011 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:44.980156 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-56db8d7945-s9np7" podUID="dad8aa30-3cc0-4626-bc03-155d60236d38" containerName="authorino" containerID="cri-o://9c374d89475cb6e7d421e92ce6ac6e0845827aa69766395d7b52663a3e207bc8" gracePeriod=30 Jul 09 19:04:45.243800 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.243775 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56db8d7945-s9np7" Jul 09 19:04:45.400537 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.400443 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dad8aa30-3cc0-4626-bc03-155d60236d38-tls-cert\") pod \"dad8aa30-3cc0-4626-bc03-155d60236d38\" (UID: \"dad8aa30-3cc0-4626-bc03-155d60236d38\") " Jul 09 19:04:45.400711 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.400550 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ww6n\" (UniqueName: \"kubernetes.io/projected/dad8aa30-3cc0-4626-bc03-155d60236d38-kube-api-access-8ww6n\") pod \"dad8aa30-3cc0-4626-bc03-155d60236d38\" (UID: \"dad8aa30-3cc0-4626-bc03-155d60236d38\") " Jul 09 19:04:45.402593 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.402559 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad8aa30-3cc0-4626-bc03-155d60236d38-kube-api-access-8ww6n" (OuterVolumeSpecName: "kube-api-access-8ww6n") pod "dad8aa30-3cc0-4626-bc03-155d60236d38" (UID: "dad8aa30-3cc0-4626-bc03-155d60236d38"). InnerVolumeSpecName "kube-api-access-8ww6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:04:45.418835 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.418667 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad8aa30-3cc0-4626-bc03-155d60236d38-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "dad8aa30-3cc0-4626-bc03-155d60236d38" (UID: "dad8aa30-3cc0-4626-bc03-155d60236d38"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:04:45.502014 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.501979 2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dad8aa30-3cc0-4626-bc03-155d60236d38-tls-cert\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:04:45.502014 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.502006 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ww6n\" (UniqueName: \"kubernetes.io/projected/dad8aa30-3cc0-4626-bc03-155d60236d38-kube-api-access-8ww6n\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:04:45.845711 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.845675 2569 generic.go:356] "Generic (PLEG): container finished" podID="dad8aa30-3cc0-4626-bc03-155d60236d38" containerID="9c374d89475cb6e7d421e92ce6ac6e0845827aa69766395d7b52663a3e207bc8" exitCode=0 Jul 09 19:04:45.846153 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.845729 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56db8d7945-s9np7" Jul 09 19:04:45.846153 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.845762 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56db8d7945-s9np7" event={"ID":"dad8aa30-3cc0-4626-bc03-155d60236d38","Type":"ContainerDied","Data":"9c374d89475cb6e7d421e92ce6ac6e0845827aa69766395d7b52663a3e207bc8"} Jul 09 19:04:45.846153 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.845837 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56db8d7945-s9np7" event={"ID":"dad8aa30-3cc0-4626-bc03-155d60236d38","Type":"ContainerDied","Data":"9046b11a1ad336792b81f3015ead193feecab748732fa4ac4429c109f932c211"} Jul 09 19:04:45.846153 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.845865 2569 scope.go:117] "RemoveContainer" containerID="9c374d89475cb6e7d421e92ce6ac6e0845827aa69766395d7b52663a3e207bc8" Jul 09 19:04:45.855028 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.854994 2569 scope.go:117] "RemoveContainer" containerID="9c374d89475cb6e7d421e92ce6ac6e0845827aa69766395d7b52663a3e207bc8" Jul 09 19:04:45.855360 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:04:45.855331 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c374d89475cb6e7d421e92ce6ac6e0845827aa69766395d7b52663a3e207bc8\": container with ID starting with 9c374d89475cb6e7d421e92ce6ac6e0845827aa69766395d7b52663a3e207bc8 not found: ID does not exist" containerID="9c374d89475cb6e7d421e92ce6ac6e0845827aa69766395d7b52663a3e207bc8" Jul 09 19:04:45.855437 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.855367 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c374d89475cb6e7d421e92ce6ac6e0845827aa69766395d7b52663a3e207bc8"} err="failed to get container status \"9c374d89475cb6e7d421e92ce6ac6e0845827aa69766395d7b52663a3e207bc8\": rpc error: code = NotFound desc = could not find container \"9c374d89475cb6e7d421e92ce6ac6e0845827aa69766395d7b52663a3e207bc8\": container with ID starting with 9c374d89475cb6e7d421e92ce6ac6e0845827aa69766395d7b52663a3e207bc8 not found: ID does not exist" Jul 09 19:04:45.866228 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.866198 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56db8d7945-s9np7"] Jul 09 19:04:45.871776 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:45.871754 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56db8d7945-s9np7"] Jul 09 19:04:47.694920 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:04:47.694889 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dad8aa30-3cc0-4626-bc03-155d60236d38" path="/var/lib/kubelet/pods/dad8aa30-3cc0-4626-bc03-155d60236d38/volumes" Jul 09 19:08:35.642688 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:08:35.642664 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 19:08:35.643202 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:08:35.642782 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 19:13:35.673778 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:13:35.673742 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 19:13:35.674344 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:13:35.674181 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 19:18:35.705251 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:18:35.705169 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 19:18:35.709209 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:18:35.709188 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 19:21:37.848259 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.848226 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk"] Jul 09 19:21:37.848766 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.848701 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad8aa30-3cc0-4626-bc03-155d60236d38" containerName="authorino" Jul 09 19:21:37.848766 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.848719 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad8aa30-3cc0-4626-bc03-155d60236d38" containerName="authorino" Jul 09 19:21:37.848889 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.848829 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dad8aa30-3cc0-4626-bc03-155d60236d38" containerName="authorino" Jul 09 19:21:37.851804 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.851784 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.855106 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.855082 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bb59c5dc\"/\"openshift-service-ca.crt\"" Jul 09 19:21:37.855234 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.855118 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bb59c5dc\"/\"kube-root-ca.crt\"" Jul 09 19:21:37.855234 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.855087 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bb59c5dc\"/\"default-dockercfg-5bvq8\"" Jul 09 19:21:37.855234 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.855085 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bb59c5dc\"/\"e2e-lifecycle-model-bb59c5dc-kserve-self-signed-certs\"" Jul 09 19:21:37.859863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.859335 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-tmp-dir\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.859863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.859386 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-home\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.859863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.859422 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-dshm\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.859863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.859473 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gxtd\" (UniqueName: \"kubernetes.io/projected/224ebd40-a789-4d16-9466-c8d6b139fed2-kube-api-access-8gxtd\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.859863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.859514 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-model-cache\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.859863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.859588 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/224ebd40-a789-4d16-9466-c8d6b139fed2-tls-certs\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.860654 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.860634 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk"] Jul 09 19:21:37.960658 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.960624 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-model-cache\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.960853 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.960678 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/224ebd40-a789-4d16-9466-c8d6b139fed2-tls-certs\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.960853 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.960701 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-tmp-dir\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.960853 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.960725 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-home\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.960853 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.960744 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-dshm\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.960853 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.960785 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gxtd\" (UniqueName: \"kubernetes.io/projected/224ebd40-a789-4d16-9466-c8d6b139fed2-kube-api-access-8gxtd\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.961177 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.961111 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-model-cache\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.961177 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.961140 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-tmp-dir\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.961246 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.961195 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-home\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.962964 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.962935 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-dshm\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.963269 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.963250 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/224ebd40-a789-4d16-9466-c8d6b139fed2-tls-certs\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:37.968699 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:37.968678 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gxtd\" (UniqueName: \"kubernetes.io/projected/224ebd40-a789-4d16-9466-c8d6b139fed2-kube-api-access-8gxtd\") pod \"e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:38.161707 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:38.161615 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:38.297557 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:38.297526 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk"] Jul 09 19:21:38.299228 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:21:38.299201 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod224ebd40_a789_4d16_9466_c8d6b139fed2.slice/crio-504ca6a660dd071bac38531bca82372579d5ea5fd00bed96363947d969e6ebab WatchSource:0}: Error finding container 504ca6a660dd071bac38531bca82372579d5ea5fd00bed96363947d969e6ebab: Status 404 returned error can't find the container with id 504ca6a660dd071bac38531bca82372579d5ea5fd00bed96363947d969e6ebab Jul 09 19:21:38.920045 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:38.920013 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" event={"ID":"224ebd40-a789-4d16-9466-c8d6b139fed2","Type":"ContainerStarted","Data":"f7da06f1ca8b63bd94f888d5bfc6a974c1d44be2b8964fca25735804fdd22049"} Jul 09 19:21:38.920045 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:38.920053 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" event={"ID":"224ebd40-a789-4d16-9466-c8d6b139fed2","Type":"ContainerStarted","Data":"504ca6a660dd071bac38531bca82372579d5ea5fd00bed96363947d969e6ebab"} Jul 09 19:21:38.937986 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:38.937927 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" podStartSLOduration=1.9379127760000001 podStartE2EDuration="1.937912776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:21:38.935866542 +0000 UTC m=+1683.761029584" watchObservedRunningTime="2026-07-09 19:21:38.937912776 +0000 UTC m=+1683.763075807" Jul 09 19:21:48.162705 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:48.162653 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:48.163199 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:48.162728 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:48.175274 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:48.175247 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:21:48.978588 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:21:48.978560 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:22:09.341514 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:09.341437 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk"] Jul 09 19:22:24.361998 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.361909 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" podUID="224ebd40-a789-4d16-9466-c8d6b139fed2" containerName="main" containerID="cri-o://f7da06f1ca8b63bd94f888d5bfc6a974c1d44be2b8964fca25735804fdd22049" gracePeriod=45 Jul 09 19:22:24.607752 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.607730 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:22:24.670159 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.670083 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-model-cache\") pod \"224ebd40-a789-4d16-9466-c8d6b139fed2\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " Jul 09 19:22:24.670308 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.670179 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-tmp-dir\") pod \"224ebd40-a789-4d16-9466-c8d6b139fed2\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " Jul 09 19:22:24.670308 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.670210 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-dshm\") pod \"224ebd40-a789-4d16-9466-c8d6b139fed2\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " Jul 09 19:22:24.670308 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.670249 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/224ebd40-a789-4d16-9466-c8d6b139fed2-tls-certs\") pod \"224ebd40-a789-4d16-9466-c8d6b139fed2\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " Jul 09 19:22:24.670308 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.670298 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gxtd\" (UniqueName: \"kubernetes.io/projected/224ebd40-a789-4d16-9466-c8d6b139fed2-kube-api-access-8gxtd\") pod \"224ebd40-a789-4d16-9466-c8d6b139fed2\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " Jul 09 19:22:24.670538 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.670326 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-home\") pod \"224ebd40-a789-4d16-9466-c8d6b139fed2\" (UID: \"224ebd40-a789-4d16-9466-c8d6b139fed2\") " Jul 09 19:22:24.670538 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.670374 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-model-cache" (OuterVolumeSpecName: "model-cache") pod "224ebd40-a789-4d16-9466-c8d6b139fed2" (UID: "224ebd40-a789-4d16-9466-c8d6b139fed2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:22:24.670655 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.670587 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-model-cache\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:22:24.670655 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.670586 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-home" (OuterVolumeSpecName: "home") pod "224ebd40-a789-4d16-9466-c8d6b139fed2" (UID: "224ebd40-a789-4d16-9466-c8d6b139fed2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:22:24.670801 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.670776 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "224ebd40-a789-4d16-9466-c8d6b139fed2" (UID: "224ebd40-a789-4d16-9466-c8d6b139fed2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:22:24.672990 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.672934 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/224ebd40-a789-4d16-9466-c8d6b139fed2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "224ebd40-a789-4d16-9466-c8d6b139fed2" (UID: "224ebd40-a789-4d16-9466-c8d6b139fed2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:22:24.672990 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.672959 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224ebd40-a789-4d16-9466-c8d6b139fed2-kube-api-access-8gxtd" (OuterVolumeSpecName: "kube-api-access-8gxtd") pod "224ebd40-a789-4d16-9466-c8d6b139fed2" (UID: "224ebd40-a789-4d16-9466-c8d6b139fed2"). InnerVolumeSpecName "kube-api-access-8gxtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:22:24.673139 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.672962 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-dshm" (OuterVolumeSpecName: "dshm") pod "224ebd40-a789-4d16-9466-c8d6b139fed2" (UID: "224ebd40-a789-4d16-9466-c8d6b139fed2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:22:24.771240 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.771203 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-tmp-dir\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:22:24.771240 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.771235 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-dshm\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:22:24.771240 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.771243 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/224ebd40-a789-4d16-9466-c8d6b139fed2-tls-certs\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:22:24.771240 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.771252 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gxtd\" (UniqueName: \"kubernetes.io/projected/224ebd40-a789-4d16-9466-c8d6b139fed2-kube-api-access-8gxtd\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:22:24.771509 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:24.771260 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/224ebd40-a789-4d16-9466-c8d6b139fed2-home\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:22:25.147036 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:25.146999 2569 generic.go:356] "Generic (PLEG): container finished" podID="224ebd40-a789-4d16-9466-c8d6b139fed2" containerID="f7da06f1ca8b63bd94f888d5bfc6a974c1d44be2b8964fca25735804fdd22049" exitCode=0 Jul 09 19:22:25.147287 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:25.147042 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" Jul 09 19:22:25.147287 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:25.147062 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" event={"ID":"224ebd40-a789-4d16-9466-c8d6b139fed2","Type":"ContainerDied","Data":"f7da06f1ca8b63bd94f888d5bfc6a974c1d44be2b8964fca25735804fdd22049"} Jul 09 19:22:25.147287 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:25.147095 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk" event={"ID":"224ebd40-a789-4d16-9466-c8d6b139fed2","Type":"ContainerDied","Data":"504ca6a660dd071bac38531bca82372579d5ea5fd00bed96363947d969e6ebab"} Jul 09 19:22:25.147287 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:25.147111 2569 scope.go:117] "RemoveContainer" containerID="f7da06f1ca8b63bd94f888d5bfc6a974c1d44be2b8964fca25735804fdd22049" Jul 09 19:22:25.156402 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:25.156384 2569 scope.go:117] "RemoveContainer" containerID="f7da06f1ca8b63bd94f888d5bfc6a974c1d44be2b8964fca25735804fdd22049" Jul 09 19:22:25.156673 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:22:25.156654 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7da06f1ca8b63bd94f888d5bfc6a974c1d44be2b8964fca25735804fdd22049\": container with ID starting with f7da06f1ca8b63bd94f888d5bfc6a974c1d44be2b8964fca25735804fdd22049 not found: ID does not exist" containerID="f7da06f1ca8b63bd94f888d5bfc6a974c1d44be2b8964fca25735804fdd22049" Jul 09 19:22:25.156745 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:25.156681 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7da06f1ca8b63bd94f888d5bfc6a974c1d44be2b8964fca25735804fdd22049"} err="failed to get container status \"f7da06f1ca8b63bd94f888d5bfc6a974c1d44be2b8964fca25735804fdd22049\": rpc error: code = NotFound desc = could not find container \"f7da06f1ca8b63bd94f888d5bfc6a974c1d44be2b8964fca25735804fdd22049\": container with ID starting with f7da06f1ca8b63bd94f888d5bfc6a974c1d44be2b8964fca25735804fdd22049 not found: ID does not exist" Jul 09 19:22:25.168868 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:25.168840 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk"] Jul 09 19:22:25.174428 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:25.174402 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-bb59c5dc/e2e-lifecycle-model-bb59c5dc-kserve-7c5b495bc9-cmctk"] Jul 09 19:22:25.696056 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:22:25.696026 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="224ebd40-a789-4d16-9466-c8d6b139fed2" path="/var/lib/kubelet/pods/224ebd40-a789-4d16-9466-c8d6b139fed2/volumes" Jul 09 19:23:35.746426 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:23:35.746400 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 19:23:35.752693 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:23:35.752667 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 19:24:49.949165 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:49.949136 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87"] Jul 09 19:24:49.949586 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:49.949470 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="224ebd40-a789-4d16-9466-c8d6b139fed2" containerName="main" Jul 09 19:24:49.949586 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:49.949481 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="224ebd40-a789-4d16-9466-c8d6b139fed2" containerName="main" Jul 09 19:24:49.949586 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:49.949559 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="224ebd40-a789-4d16-9466-c8d6b139fed2" containerName="main" Jul 09 19:24:49.952384 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:49.952363 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:49.955676 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:49.955652 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c6e06b\"/\"default-dockercfg-r8g9w\"" Jul 09 19:24:49.956138 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:49.956118 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c6e06b\"/\"kube-root-ca.crt\"" Jul 09 19:24:49.956480 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:49.956463 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c6e06b\"/\"openshift-service-ca.crt\"" Jul 09 19:24:49.956542 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:49.956463 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c6e06b\"/\"test-model-c6e06b-kserve-self-signed-certs\"" Jul 09 19:24:49.964143 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:49.964121 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87"] Jul 09 19:24:50.053102 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.053069 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-dshm\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.053281 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.053114 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-model-cache\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.053281 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.053139 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfdvr\" (UniqueName: \"kubernetes.io/projected/7feb1119-42f7-42ad-b4bd-dd4c24265189-kube-api-access-zfdvr\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.053281 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.053191 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-home\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.053281 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.053252 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7feb1119-42f7-42ad-b4bd-dd4c24265189-tls-certs\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.053421 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.053325 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-tmp-dir\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.153878 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.153843 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-home\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.153878 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.153894 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7feb1119-42f7-42ad-b4bd-dd4c24265189-tls-certs\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.154168 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.154021 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-tmp-dir\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.154168 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.154069 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-dshm\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.154168 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.154105 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-model-cache\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.154168 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.154121 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfdvr\" (UniqueName: \"kubernetes.io/projected/7feb1119-42f7-42ad-b4bd-dd4c24265189-kube-api-access-zfdvr\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.154344 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.154320 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-home\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.154394 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.154345 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-tmp-dir\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.154542 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.154526 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-model-cache\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.156198 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.156172 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-dshm\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.156440 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.156423 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7feb1119-42f7-42ad-b4bd-dd4c24265189-tls-certs\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.162224 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.162199 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfdvr\" (UniqueName: \"kubernetes.io/projected/7feb1119-42f7-42ad-b4bd-dd4c24265189-kube-api-access-zfdvr\") pod \"test-model-c6e06b-kserve-7f7f6c75f7-qpx87\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.263917 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.263877 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:24:50.392412 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.392390 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87"] Jul 09 19:24:50.394547 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:24:50.394520 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7feb1119_42f7_42ad_b4bd_dd4c24265189.slice/crio-a3eafbe9dd8fd8ceb7720c523ef0a8a97a5bc29ee9523114488627f4e13e7734 WatchSource:0}: Error finding container a3eafbe9dd8fd8ceb7720c523ef0a8a97a5bc29ee9523114488627f4e13e7734: Status 404 returned error can't find the container with id a3eafbe9dd8fd8ceb7720c523ef0a8a97a5bc29ee9523114488627f4e13e7734 Jul 09 19:24:50.862537 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.862498 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" event={"ID":"7feb1119-42f7-42ad-b4bd-dd4c24265189","Type":"ContainerStarted","Data":"609fad16c518944c4b18e057fbed7e7678a96703cf50e96f6fdf561b1c5bb07e"} Jul 09 19:24:50.862716 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.862550 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" event={"ID":"7feb1119-42f7-42ad-b4bd-dd4c24265189","Type":"ContainerStarted","Data":"a3eafbe9dd8fd8ceb7720c523ef0a8a97a5bc29ee9523114488627f4e13e7734"} Jul 09 19:24:50.881093 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:24:50.881027 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" podStartSLOduration=1.881006886 podStartE2EDuration="1.881006886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:24:50.879046866 +0000 UTC m=+1875.704209906" watchObservedRunningTime="2026-07-09 19:24:50.881006886 +0000 UTC m=+1875.706169923" Jul 09 19:25:00.264377 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:00.264295 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:25:00.264834 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:00.264586 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:25:00.276691 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:00.276664 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:25:00.927310 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:00.927281 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:25:07.640972 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.640929 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q"] Jul 09 19:25:07.644613 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.644596 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.647197 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.647171 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-eb3d8a\"/\"openshift-service-ca.crt\"" Jul 09 19:25:07.648313 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.648294 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-eb3d8a\"/\"default-dockercfg-2sg4t\"" Jul 09 19:25:07.648374 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.648299 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-eb3d8a\"/\"test-model-eb3d8a-kserve-self-signed-certs\"" Jul 09 19:25:07.648374 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.648328 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-eb3d8a\"/\"kube-root-ca.crt\"" Jul 09 19:25:07.653556 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.653533 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q"] Jul 09 19:25:07.721611 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.721582 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20baa971-f2a9-4dda-ab4f-9a51fde82707-tls-certs\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.721611 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.721621 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-dshm\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.721863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.721640 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-tmp-dir\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.721863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.721664 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-home\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.721863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.721769 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-model-cache\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.721863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.721831 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82j69\" (UniqueName: \"kubernetes.io/projected/20baa971-f2a9-4dda-ab4f-9a51fde82707-kube-api-access-82j69\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.822652 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.822613 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20baa971-f2a9-4dda-ab4f-9a51fde82707-tls-certs\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.822863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.822661 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-dshm\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.822863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.822679 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-tmp-dir\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.822863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.822706 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-home\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.822863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.822768 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-model-cache\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.822863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.822807 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82j69\" (UniqueName: \"kubernetes.io/projected/20baa971-f2a9-4dda-ab4f-9a51fde82707-kube-api-access-82j69\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.823204 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.823175 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-home\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.823248 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.823210 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-tmp-dir\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.823284 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.823251 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-model-cache\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.825193 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.825173 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20baa971-f2a9-4dda-ab4f-9a51fde82707-tls-certs\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.825193 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.825188 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-dshm\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.832069 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.832042 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82j69\" (UniqueName: \"kubernetes.io/projected/20baa971-f2a9-4dda-ab4f-9a51fde82707-kube-api-access-82j69\") pod \"test-model-eb3d8a-kserve-5f8b9bbff-zgj8q\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:07.955845 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:07.955760 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:08.091848 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:08.091822 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q"] Jul 09 19:25:08.093832 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:25:08.093805 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20baa971_f2a9_4dda_ab4f_9a51fde82707.slice/crio-ad927a8095151f025bfba158ec33f6b7368f54376dba87ded6ff680a066e20f9 WatchSource:0}: Error finding container ad927a8095151f025bfba158ec33f6b7368f54376dba87ded6ff680a066e20f9: Status 404 returned error can't find the container with id ad927a8095151f025bfba158ec33f6b7368f54376dba87ded6ff680a066e20f9 Jul 09 19:25:08.955998 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:08.955962 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" event={"ID":"20baa971-f2a9-4dda-ab4f-9a51fde82707","Type":"ContainerStarted","Data":"60c1170d8a2911a2c54a038aa4adf8647acb8450139eefff6a94af21f4d3d565"} Jul 09 19:25:08.955998 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:08.956002 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" event={"ID":"20baa971-f2a9-4dda-ab4f-9a51fde82707","Type":"ContainerStarted","Data":"ad927a8095151f025bfba158ec33f6b7368f54376dba87ded6ff680a066e20f9"} Jul 09 19:25:08.987940 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:08.987885 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" podStartSLOduration=1.987867467 podStartE2EDuration="1.987867467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:25:08.985833311 +0000 UTC m=+1893.810996345" watchObservedRunningTime="2026-07-09 19:25:08.987867467 +0000 UTC m=+1893.813030505" Jul 09 19:25:17.956157 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:17.956113 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:17.956157 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:17.956169 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:17.968649 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:17.968617 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:18.016841 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:18.016813 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:30.431491 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:30.431451 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87"] Jul 09 19:25:31.320743 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:31.320708 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q"] Jul 09 19:25:45.451793 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.451748 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" podUID="7feb1119-42f7-42ad-b4bd-dd4c24265189" containerName="main" containerID="cri-o://609fad16c518944c4b18e057fbed7e7678a96703cf50e96f6fdf561b1c5bb07e" gracePeriod=45 Jul 09 19:25:45.698627 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.698602 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:25:45.757401 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.757374 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-tmp-dir\") pod \"7feb1119-42f7-42ad-b4bd-dd4c24265189\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " Jul 09 19:25:45.757587 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.757413 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfdvr\" (UniqueName: \"kubernetes.io/projected/7feb1119-42f7-42ad-b4bd-dd4c24265189-kube-api-access-zfdvr\") pod \"7feb1119-42f7-42ad-b4bd-dd4c24265189\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " Jul 09 19:25:45.757587 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.757438 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7feb1119-42f7-42ad-b4bd-dd4c24265189-tls-certs\") pod \"7feb1119-42f7-42ad-b4bd-dd4c24265189\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " Jul 09 19:25:45.757587 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.757474 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-model-cache\") pod \"7feb1119-42f7-42ad-b4bd-dd4c24265189\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " Jul 09 19:25:45.757587 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.757491 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-home\") pod \"7feb1119-42f7-42ad-b4bd-dd4c24265189\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " Jul 09 19:25:45.757587 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.757568 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-dshm\") pod \"7feb1119-42f7-42ad-b4bd-dd4c24265189\" (UID: \"7feb1119-42f7-42ad-b4bd-dd4c24265189\") " Jul 09 19:25:45.757868 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.757670 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7feb1119-42f7-42ad-b4bd-dd4c24265189" (UID: "7feb1119-42f7-42ad-b4bd-dd4c24265189"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:25:45.757868 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.757755 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-model-cache" (OuterVolumeSpecName: "model-cache") pod "7feb1119-42f7-42ad-b4bd-dd4c24265189" (UID: "7feb1119-42f7-42ad-b4bd-dd4c24265189"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:25:45.758039 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.757888 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-home" (OuterVolumeSpecName: "home") pod "7feb1119-42f7-42ad-b4bd-dd4c24265189" (UID: "7feb1119-42f7-42ad-b4bd-dd4c24265189"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:25:45.758039 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.757934 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-tmp-dir\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:25:45.758039 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.757968 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-model-cache\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:25:45.759761 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.759730 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-dshm" (OuterVolumeSpecName: "dshm") pod "7feb1119-42f7-42ad-b4bd-dd4c24265189" (UID: "7feb1119-42f7-42ad-b4bd-dd4c24265189"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:25:45.759913 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.759786 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7feb1119-42f7-42ad-b4bd-dd4c24265189-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7feb1119-42f7-42ad-b4bd-dd4c24265189" (UID: "7feb1119-42f7-42ad-b4bd-dd4c24265189"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:25:45.759913 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.759798 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7feb1119-42f7-42ad-b4bd-dd4c24265189-kube-api-access-zfdvr" (OuterVolumeSpecName: "kube-api-access-zfdvr") pod "7feb1119-42f7-42ad-b4bd-dd4c24265189" (UID: "7feb1119-42f7-42ad-b4bd-dd4c24265189"). InnerVolumeSpecName "kube-api-access-zfdvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:25:45.858393 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.858369 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-home\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:25:45.858393 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.858390 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7feb1119-42f7-42ad-b4bd-dd4c24265189-dshm\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:25:45.858393 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.858399 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfdvr\" (UniqueName: \"kubernetes.io/projected/7feb1119-42f7-42ad-b4bd-dd4c24265189-kube-api-access-zfdvr\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:25:45.858606 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:45.858409 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7feb1119-42f7-42ad-b4bd-dd4c24265189-tls-certs\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:25:46.161764 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.161671 2569 generic.go:356] "Generic (PLEG): container finished" podID="7feb1119-42f7-42ad-b4bd-dd4c24265189" containerID="609fad16c518944c4b18e057fbed7e7678a96703cf50e96f6fdf561b1c5bb07e" exitCode=0 Jul 09 19:25:46.161764 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.161719 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" Jul 09 19:25:46.161971 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.161755 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" event={"ID":"7feb1119-42f7-42ad-b4bd-dd4c24265189","Type":"ContainerDied","Data":"609fad16c518944c4b18e057fbed7e7678a96703cf50e96f6fdf561b1c5bb07e"} Jul 09 19:25:46.161971 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.161801 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87" event={"ID":"7feb1119-42f7-42ad-b4bd-dd4c24265189","Type":"ContainerDied","Data":"a3eafbe9dd8fd8ceb7720c523ef0a8a97a5bc29ee9523114488627f4e13e7734"} Jul 09 19:25:46.161971 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.161823 2569 scope.go:117] "RemoveContainer" containerID="609fad16c518944c4b18e057fbed7e7678a96703cf50e96f6fdf561b1c5bb07e" Jul 09 19:25:46.170863 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.170842 2569 scope.go:117] "RemoveContainer" containerID="609fad16c518944c4b18e057fbed7e7678a96703cf50e96f6fdf561b1c5bb07e" Jul 09 19:25:46.171233 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:25:46.171211 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"609fad16c518944c4b18e057fbed7e7678a96703cf50e96f6fdf561b1c5bb07e\": container with ID starting with 609fad16c518944c4b18e057fbed7e7678a96703cf50e96f6fdf561b1c5bb07e not found: ID does not exist" containerID="609fad16c518944c4b18e057fbed7e7678a96703cf50e96f6fdf561b1c5bb07e" Jul 09 19:25:46.171317 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.171245 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"609fad16c518944c4b18e057fbed7e7678a96703cf50e96f6fdf561b1c5bb07e"} err="failed to get container status \"609fad16c518944c4b18e057fbed7e7678a96703cf50e96f6fdf561b1c5bb07e\": rpc error: code = NotFound desc = could not find container \"609fad16c518944c4b18e057fbed7e7678a96703cf50e96f6fdf561b1c5bb07e\": container with ID starting with 609fad16c518944c4b18e057fbed7e7678a96703cf50e96f6fdf561b1c5bb07e not found: ID does not exist" Jul 09 19:25:46.183135 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.183108 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87"] Jul 09 19:25:46.189719 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.189700 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c6e06b/test-model-c6e06b-kserve-7f7f6c75f7-qpx87"] Jul 09 19:25:46.340922 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.340861 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" podUID="20baa971-f2a9-4dda-ab4f-9a51fde82707" containerName="main" containerID="cri-o://60c1170d8a2911a2c54a038aa4adf8647acb8450139eefff6a94af21f4d3d565" gracePeriod=45 Jul 09 19:25:46.580100 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.580073 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:46.665747 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.665719 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20baa971-f2a9-4dda-ab4f-9a51fde82707-tls-certs\") pod \"20baa971-f2a9-4dda-ab4f-9a51fde82707\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " Jul 09 19:25:46.665911 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.665761 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-tmp-dir\") pod \"20baa971-f2a9-4dda-ab4f-9a51fde82707\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " Jul 09 19:25:46.665911 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.665778 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-model-cache\") pod \"20baa971-f2a9-4dda-ab4f-9a51fde82707\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " Jul 09 19:25:46.666094 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.665939 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82j69\" (UniqueName: \"kubernetes.io/projected/20baa971-f2a9-4dda-ab4f-9a51fde82707-kube-api-access-82j69\") pod \"20baa971-f2a9-4dda-ab4f-9a51fde82707\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " Jul 09 19:25:46.666094 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.666007 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-dshm\") pod \"20baa971-f2a9-4dda-ab4f-9a51fde82707\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " Jul 09 19:25:46.666094 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.666046 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-model-cache" (OuterVolumeSpecName: "model-cache") pod "20baa971-f2a9-4dda-ab4f-9a51fde82707" (UID: "20baa971-f2a9-4dda-ab4f-9a51fde82707"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:25:46.666094 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.666060 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20baa971-f2a9-4dda-ab4f-9a51fde82707" (UID: "20baa971-f2a9-4dda-ab4f-9a51fde82707"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:25:46.666094 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.666075 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-home\") pod \"20baa971-f2a9-4dda-ab4f-9a51fde82707\" (UID: \"20baa971-f2a9-4dda-ab4f-9a51fde82707\") " Jul 09 19:25:46.666360 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.666328 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-tmp-dir\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:25:46.666360 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.666344 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-model-cache\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:25:46.666360 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.666348 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-home" (OuterVolumeSpecName: "home") pod "20baa971-f2a9-4dda-ab4f-9a51fde82707" (UID: "20baa971-f2a9-4dda-ab4f-9a51fde82707"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:25:46.667998 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.667976 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20baa971-f2a9-4dda-ab4f-9a51fde82707-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20baa971-f2a9-4dda-ab4f-9a51fde82707" (UID: "20baa971-f2a9-4dda-ab4f-9a51fde82707"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:25:46.668104 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.668082 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20baa971-f2a9-4dda-ab4f-9a51fde82707-kube-api-access-82j69" (OuterVolumeSpecName: "kube-api-access-82j69") pod "20baa971-f2a9-4dda-ab4f-9a51fde82707" (UID: "20baa971-f2a9-4dda-ab4f-9a51fde82707"). InnerVolumeSpecName "kube-api-access-82j69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:25:46.668153 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.668129 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-dshm" (OuterVolumeSpecName: "dshm") pod "20baa971-f2a9-4dda-ab4f-9a51fde82707" (UID: "20baa971-f2a9-4dda-ab4f-9a51fde82707"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:25:46.767451 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.767417 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20baa971-f2a9-4dda-ab4f-9a51fde82707-tls-certs\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:25:46.767451 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.767446 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82j69\" (UniqueName: \"kubernetes.io/projected/20baa971-f2a9-4dda-ab4f-9a51fde82707-kube-api-access-82j69\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:25:46.767451 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.767456 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-dshm\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:25:46.767451 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:46.767464 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20baa971-f2a9-4dda-ab4f-9a51fde82707-home\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:25:47.168469 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:47.168367 2569 generic.go:356] "Generic (PLEG): container finished" podID="20baa971-f2a9-4dda-ab4f-9a51fde82707" containerID="60c1170d8a2911a2c54a038aa4adf8647acb8450139eefff6a94af21f4d3d565" exitCode=0 Jul 09 19:25:47.168469 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:47.168417 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" Jul 09 19:25:47.168469 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:47.168456 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" event={"ID":"20baa971-f2a9-4dda-ab4f-9a51fde82707","Type":"ContainerDied","Data":"60c1170d8a2911a2c54a038aa4adf8647acb8450139eefff6a94af21f4d3d565"} Jul 09 19:25:47.168714 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:47.168497 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q" event={"ID":"20baa971-f2a9-4dda-ab4f-9a51fde82707","Type":"ContainerDied","Data":"ad927a8095151f025bfba158ec33f6b7368f54376dba87ded6ff680a066e20f9"} Jul 09 19:25:47.168714 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:47.168515 2569 scope.go:117] "RemoveContainer" containerID="60c1170d8a2911a2c54a038aa4adf8647acb8450139eefff6a94af21f4d3d565" Jul 09 19:25:47.177720 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:47.177702 2569 scope.go:117] "RemoveContainer" containerID="60c1170d8a2911a2c54a038aa4adf8647acb8450139eefff6a94af21f4d3d565" Jul 09 19:25:47.177993 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:25:47.177976 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c1170d8a2911a2c54a038aa4adf8647acb8450139eefff6a94af21f4d3d565\": container with ID starting with 60c1170d8a2911a2c54a038aa4adf8647acb8450139eefff6a94af21f4d3d565 not found: ID does not exist" containerID="60c1170d8a2911a2c54a038aa4adf8647acb8450139eefff6a94af21f4d3d565" Jul 09 19:25:47.178053 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:47.177999 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c1170d8a2911a2c54a038aa4adf8647acb8450139eefff6a94af21f4d3d565"} err="failed to get container status \"60c1170d8a2911a2c54a038aa4adf8647acb8450139eefff6a94af21f4d3d565\": rpc error: code = NotFound desc = could not find container \"60c1170d8a2911a2c54a038aa4adf8647acb8450139eefff6a94af21f4d3d565\": container with ID starting with 60c1170d8a2911a2c54a038aa4adf8647acb8450139eefff6a94af21f4d3d565 not found: ID does not exist" Jul 09 19:25:47.190543 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:47.190519 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q"] Jul 09 19:25:47.192477 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:47.192454 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-eb3d8a/test-model-eb3d8a-kserve-5f8b9bbff-zgj8q"] Jul 09 19:25:47.694399 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:47.694367 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20baa971-f2a9-4dda-ab4f-9a51fde82707" path="/var/lib/kubelet/pods/20baa971-f2a9-4dda-ab4f-9a51fde82707/volumes" Jul 09 19:25:47.694794 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:25:47.694737 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7feb1119-42f7-42ad-b4bd-dd4c24265189" path="/var/lib/kubelet/pods/7feb1119-42f7-42ad-b4bd-dd4c24265189/volumes" Jul 09 19:27:56.648595 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.648563 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7"] Jul 09 19:27:56.649101 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.648916 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20baa971-f2a9-4dda-ab4f-9a51fde82707" containerName="main" Jul 09 19:27:56.649101 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.648928 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="20baa971-f2a9-4dda-ab4f-9a51fde82707" containerName="main" Jul 09 19:27:56.649101 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.648969 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7feb1119-42f7-42ad-b4bd-dd4c24265189" containerName="main" Jul 09 19:27:56.649101 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.648976 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7feb1119-42f7-42ad-b4bd-dd4c24265189" containerName="main" Jul 09 19:27:56.649101 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.649035 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="20baa971-f2a9-4dda-ab4f-9a51fde82707" containerName="main" Jul 09 19:27:56.649101 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.649045 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7feb1119-42f7-42ad-b4bd-dd4c24265189" containerName="main" Jul 09 19:27:56.652135 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.652118 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.655684 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.655663 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5d8f8f\"/\"kube-root-ca.crt\"" Jul 09 19:27:56.655845 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.655696 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5d8f8f\"/\"default-dockercfg-crx4n\"" Jul 09 19:27:56.655845 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.655696 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5d8f8f\"/\"auth-test-model-5d8f8f-kserve-self-signed-certs\"" Jul 09 19:27:56.655845 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.655775 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5d8f8f\"/\"openshift-service-ca.crt\"" Jul 09 19:27:56.664776 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.664754 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7"] Jul 09 19:27:56.720294 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.720260 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-home\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.720459 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.720334 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-dshm\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.720459 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.720389 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-tmp-dir\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.720459 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.720431 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9ff940-bd8b-45e0-99f8-d0215a0f631f-tls-certs\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.720588 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.720460 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755wb\" (UniqueName: \"kubernetes.io/projected/da9ff940-bd8b-45e0-99f8-d0215a0f631f-kube-api-access-755wb\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.720588 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.720486 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-model-cache\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.821922 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.821884 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-dshm\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.822263 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.822243 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-tmp-dir\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.822668 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.822637 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-tmp-dir\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.823030 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.823006 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9ff940-bd8b-45e0-99f8-d0215a0f631f-tls-certs\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.823233 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.823202 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755wb\" (UniqueName: \"kubernetes.io/projected/da9ff940-bd8b-45e0-99f8-d0215a0f631f-kube-api-access-755wb\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.823351 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.823252 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-model-cache\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.823351 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.823282 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-home\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.823587 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.823569 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-home\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.823674 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.823654 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-model-cache\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.824452 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.824436 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-dshm\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.825273 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.825255 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9ff940-bd8b-45e0-99f8-d0215a0f631f-tls-certs\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.830623 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.830602 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-755wb\" (UniqueName: \"kubernetes.io/projected/da9ff940-bd8b-45e0-99f8-d0215a0f631f-kube-api-access-755wb\") pod \"auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:56.961339 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:56.961313 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:27:57.091633 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:57.091608 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7"] Jul 09 19:27:57.093710 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:27:57.093683 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda9ff940_bd8b_45e0_99f8_d0215a0f631f.slice/crio-082997fffd1ff0ec37782ec85685231372cc0087135975024e548b609304215a WatchSource:0}: Error finding container 082997fffd1ff0ec37782ec85685231372cc0087135975024e548b609304215a: Status 404 returned error can't find the container with id 082997fffd1ff0ec37782ec85685231372cc0087135975024e548b609304215a Jul 09 19:27:57.817810 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:57.817775 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" event={"ID":"da9ff940-bd8b-45e0-99f8-d0215a0f631f","Type":"ContainerStarted","Data":"77277451f3ef70873040361b6931be563369c6a4532071b17ab1b2cfa6b5131d"} Jul 09 19:27:57.818229 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:57.817821 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" event={"ID":"da9ff940-bd8b-45e0-99f8-d0215a0f631f","Type":"ContainerStarted","Data":"082997fffd1ff0ec37782ec85685231372cc0087135975024e548b609304215a"} Jul 09 19:27:57.836672 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:27:57.836625 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" podStartSLOduration=1.836612887 podStartE2EDuration="1.836612887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:27:57.835088898 +0000 UTC m=+2062.660251936" watchObservedRunningTime="2026-07-09 19:27:57.836612887 +0000 UTC m=+2062.661775923" Jul 09 19:28:06.961647 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:06.961610 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:28:06.962060 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:06.961661 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:28:06.974187 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:06.974164 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:28:07.879882 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:07.879852 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:28:20.347635 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.347602 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq"] Jul 09 19:28:20.351128 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.351110 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.354648 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.354618 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b20a51\"/\"default-dockercfg-v4pq8\"" Jul 09 19:28:20.354779 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.354710 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b20a51\"/\"auth-test-model-b20a51-kserve-self-signed-certs\"" Jul 09 19:28:20.354779 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.354726 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b20a51\"/\"kube-root-ca.crt\"" Jul 09 19:28:20.354779 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.354719 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b20a51\"/\"openshift-service-ca.crt\"" Jul 09 19:28:20.361251 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.361223 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq"] Jul 09 19:28:20.536844 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.536809 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-model-cache\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.537048 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.536854 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-tmp-dir\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.537048 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.536885 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-home\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.537048 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.536983 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lthr\" (UniqueName: \"kubernetes.io/projected/6abe0bdd-422b-422f-a638-901fc06061e4-kube-api-access-2lthr\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.537048 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.537040 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-dshm\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.537203 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.537089 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6abe0bdd-422b-422f-a638-901fc06061e4-tls-certs\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.638599 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.638510 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-tmp-dir\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.638599 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.638557 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-home\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.638599 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.638590 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lthr\" (UniqueName: \"kubernetes.io/projected/6abe0bdd-422b-422f-a638-901fc06061e4-kube-api-access-2lthr\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.638860 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.638618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-dshm\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.638860 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.638645 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6abe0bdd-422b-422f-a638-901fc06061e4-tls-certs\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.638860 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.638677 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-model-cache\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.639055 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.639013 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-tmp-dir\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.639055 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.639031 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-home\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.639157 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.639072 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-model-cache\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.640987 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.640958 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-dshm\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.641196 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.641181 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6abe0bdd-422b-422f-a638-901fc06061e4-tls-certs\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.647600 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.647577 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lthr\" (UniqueName: \"kubernetes.io/projected/6abe0bdd-422b-422f-a638-901fc06061e4-kube-api-access-2lthr\") pod \"auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.663472 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.663448 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:20.795081 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.795055 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq"] Jul 09 19:28:20.797023 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:28:20.796997 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6abe0bdd_422b_422f_a638_901fc06061e4.slice/crio-12f06582e848d6352c37dedb56be1ea631b0cfc317d5943783ab910b254721be WatchSource:0}: Error finding container 12f06582e848d6352c37dedb56be1ea631b0cfc317d5943783ab910b254721be: Status 404 returned error can't find the container with id 12f06582e848d6352c37dedb56be1ea631b0cfc317d5943783ab910b254721be Jul 09 19:28:20.934609 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.934517 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" event={"ID":"6abe0bdd-422b-422f-a638-901fc06061e4","Type":"ContainerStarted","Data":"22ebd21236f31d3d8557f4d643ada0a786bc65e62a9c61978d5e095810277498"} Jul 09 19:28:20.934609 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.934571 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" event={"ID":"6abe0bdd-422b-422f-a638-901fc06061e4","Type":"ContainerStarted","Data":"12f06582e848d6352c37dedb56be1ea631b0cfc317d5943783ab910b254721be"} Jul 09 19:28:20.952801 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:20.952749 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" podStartSLOduration=0.952733309 podStartE2EDuration="952.733309ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:28:20.951071052 +0000 UTC m=+2085.776234090" watchObservedRunningTime="2026-07-09 19:28:20.952733309 +0000 UTC m=+2085.777896345" Jul 09 19:28:30.664511 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:30.664472 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:30.664511 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:30.664518 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:30.677531 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:30.677510 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:30.994187 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:30.994160 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:28:35.778476 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:35.778447 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 19:28:35.785454 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:28:35.785427 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 19:30:24.323200 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:24.323165 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7"] Jul 09 19:30:25.528063 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:25.528019 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq"] Jul 09 19:30:39.343528 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.343473 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" podUID="da9ff940-bd8b-45e0-99f8-d0215a0f631f" containerName="main" containerID="cri-o://77277451f3ef70873040361b6931be563369c6a4532071b17ab1b2cfa6b5131d" gracePeriod=45 Jul 09 19:30:39.592483 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.592451 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:30:39.623296 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.623217 2569 generic.go:356] "Generic (PLEG): container finished" podID="da9ff940-bd8b-45e0-99f8-d0215a0f631f" containerID="77277451f3ef70873040361b6931be563369c6a4532071b17ab1b2cfa6b5131d" exitCode=0 Jul 09 19:30:39.623296 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.623259 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" Jul 09 19:30:39.623296 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.623268 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" event={"ID":"da9ff940-bd8b-45e0-99f8-d0215a0f631f","Type":"ContainerDied","Data":"77277451f3ef70873040361b6931be563369c6a4532071b17ab1b2cfa6b5131d"} Jul 09 19:30:39.623538 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.623303 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7" event={"ID":"da9ff940-bd8b-45e0-99f8-d0215a0f631f","Type":"ContainerDied","Data":"082997fffd1ff0ec37782ec85685231372cc0087135975024e548b609304215a"} Jul 09 19:30:39.623538 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.623322 2569 scope.go:117] "RemoveContainer" containerID="77277451f3ef70873040361b6931be563369c6a4532071b17ab1b2cfa6b5131d" Jul 09 19:30:39.632988 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.632970 2569 scope.go:117] "RemoveContainer" containerID="77277451f3ef70873040361b6931be563369c6a4532071b17ab1b2cfa6b5131d" Jul 09 19:30:39.633258 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:30:39.633239 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77277451f3ef70873040361b6931be563369c6a4532071b17ab1b2cfa6b5131d\": container with ID starting with 77277451f3ef70873040361b6931be563369c6a4532071b17ab1b2cfa6b5131d not found: ID does not exist" containerID="77277451f3ef70873040361b6931be563369c6a4532071b17ab1b2cfa6b5131d" Jul 09 19:30:39.633314 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.633271 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77277451f3ef70873040361b6931be563369c6a4532071b17ab1b2cfa6b5131d"} err="failed to get container status \"77277451f3ef70873040361b6931be563369c6a4532071b17ab1b2cfa6b5131d\": rpc error: code = NotFound desc = could not find container \"77277451f3ef70873040361b6931be563369c6a4532071b17ab1b2cfa6b5131d\": container with ID starting with 77277451f3ef70873040361b6931be563369c6a4532071b17ab1b2cfa6b5131d not found: ID does not exist" Jul 09 19:30:39.664195 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.664168 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-tmp-dir\") pod \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " Jul 09 19:30:39.664351 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.664208 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-home\") pod \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " Jul 09 19:30:39.664351 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.664259 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9ff940-bd8b-45e0-99f8-d0215a0f631f-tls-certs\") pod \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " Jul 09 19:30:39.664351 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.664281 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-model-cache\") pod \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " Jul 09 19:30:39.664351 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.664317 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-755wb\" (UniqueName: \"kubernetes.io/projected/da9ff940-bd8b-45e0-99f8-d0215a0f631f-kube-api-access-755wb\") pod \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " Jul 09 19:30:39.664351 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.664336 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-dshm\") pod \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\" (UID: \"da9ff940-bd8b-45e0-99f8-d0215a0f631f\") " Jul 09 19:30:39.664611 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.664466 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da9ff940-bd8b-45e0-99f8-d0215a0f631f" (UID: "da9ff940-bd8b-45e0-99f8-d0215a0f631f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:30:39.664611 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.664518 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-home" (OuterVolumeSpecName: "home") pod "da9ff940-bd8b-45e0-99f8-d0215a0f631f" (UID: "da9ff940-bd8b-45e0-99f8-d0215a0f631f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:30:39.664611 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.664561 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-model-cache" (OuterVolumeSpecName: "model-cache") pod "da9ff940-bd8b-45e0-99f8-d0215a0f631f" (UID: "da9ff940-bd8b-45e0-99f8-d0215a0f631f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:30:39.664750 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.664733 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-model-cache\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:30:39.664806 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.664751 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-tmp-dir\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:30:39.664806 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.664760 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-home\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:30:39.666523 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.666500 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9ff940-bd8b-45e0-99f8-d0215a0f631f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da9ff940-bd8b-45e0-99f8-d0215a0f631f" (UID: "da9ff940-bd8b-45e0-99f8-d0215a0f631f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:30:39.666598 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.666530 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9ff940-bd8b-45e0-99f8-d0215a0f631f-kube-api-access-755wb" (OuterVolumeSpecName: "kube-api-access-755wb") pod "da9ff940-bd8b-45e0-99f8-d0215a0f631f" (UID: "da9ff940-bd8b-45e0-99f8-d0215a0f631f"). InnerVolumeSpecName "kube-api-access-755wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:30:39.666598 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.666585 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-dshm" (OuterVolumeSpecName: "dshm") pod "da9ff940-bd8b-45e0-99f8-d0215a0f631f" (UID: "da9ff940-bd8b-45e0-99f8-d0215a0f631f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:30:39.766727 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.766472 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9ff940-bd8b-45e0-99f8-d0215a0f631f-tls-certs\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:30:39.766727 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.766503 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-755wb\" (UniqueName: \"kubernetes.io/projected/da9ff940-bd8b-45e0-99f8-d0215a0f631f-kube-api-access-755wb\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:30:39.766727 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.766517 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9ff940-bd8b-45e0-99f8-d0215a0f631f-dshm\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:30:39.938903 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.938819 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7"] Jul 09 19:30:39.944350 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:39.944322 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/auth-test-model-5d8f8f-kserve-785b9c9f99-lpvn7"] Jul 09 19:30:40.548392 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.548344 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" podUID="6abe0bdd-422b-422f-a638-901fc06061e4" containerName="main" containerID="cri-o://22ebd21236f31d3d8557f4d643ada0a786bc65e62a9c61978d5e095810277498" gracePeriod=45 Jul 09 19:30:40.802207 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.802140 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:30:40.877759 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.877724 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-model-cache\") pod \"6abe0bdd-422b-422f-a638-901fc06061e4\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " Jul 09 19:30:40.877759 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.877769 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-dshm\") pod \"6abe0bdd-422b-422f-a638-901fc06061e4\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " Jul 09 19:30:40.878080 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.877819 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6abe0bdd-422b-422f-a638-901fc06061e4-tls-certs\") pod \"6abe0bdd-422b-422f-a638-901fc06061e4\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " Jul 09 19:30:40.878080 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.877895 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-tmp-dir\") pod \"6abe0bdd-422b-422f-a638-901fc06061e4\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " Jul 09 19:30:40.878080 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.877972 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lthr\" (UniqueName: \"kubernetes.io/projected/6abe0bdd-422b-422f-a638-901fc06061e4-kube-api-access-2lthr\") pod \"6abe0bdd-422b-422f-a638-901fc06061e4\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " Jul 09 19:30:40.878080 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.878012 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "6abe0bdd-422b-422f-a638-901fc06061e4" (UID: "6abe0bdd-422b-422f-a638-901fc06061e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:30:40.878080 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.878025 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-home\") pod \"6abe0bdd-422b-422f-a638-901fc06061e4\" (UID: \"6abe0bdd-422b-422f-a638-901fc06061e4\") " Jul 09 19:30:40.878366 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.878185 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6abe0bdd-422b-422f-a638-901fc06061e4" (UID: "6abe0bdd-422b-422f-a638-901fc06061e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:30:40.878366 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.878242 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-home" (OuterVolumeSpecName: "home") pod "6abe0bdd-422b-422f-a638-901fc06061e4" (UID: "6abe0bdd-422b-422f-a638-901fc06061e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:30:40.878479 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.878364 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-tmp-dir\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:30:40.878479 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.878379 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-home\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:30:40.878479 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.878388 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-model-cache\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:30:40.880179 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.880151 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6abe0bdd-422b-422f-a638-901fc06061e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6abe0bdd-422b-422f-a638-901fc06061e4" (UID: "6abe0bdd-422b-422f-a638-901fc06061e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:30:40.880297 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.880190 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-dshm" (OuterVolumeSpecName: "dshm") pod "6abe0bdd-422b-422f-a638-901fc06061e4" (UID: "6abe0bdd-422b-422f-a638-901fc06061e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:30:40.880297 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.880217 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abe0bdd-422b-422f-a638-901fc06061e4-kube-api-access-2lthr" (OuterVolumeSpecName: "kube-api-access-2lthr") pod "6abe0bdd-422b-422f-a638-901fc06061e4" (UID: "6abe0bdd-422b-422f-a638-901fc06061e4"). InnerVolumeSpecName "kube-api-access-2lthr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:30:40.979727 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.979695 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lthr\" (UniqueName: \"kubernetes.io/projected/6abe0bdd-422b-422f-a638-901fc06061e4-kube-api-access-2lthr\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:30:40.979727 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.979724 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abe0bdd-422b-422f-a638-901fc06061e4-dshm\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:30:40.979727 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:40.979734 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6abe0bdd-422b-422f-a638-901fc06061e4-tls-certs\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:30:41.641169 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:41.641133 2569 generic.go:356] "Generic (PLEG): container finished" podID="6abe0bdd-422b-422f-a638-901fc06061e4" containerID="22ebd21236f31d3d8557f4d643ada0a786bc65e62a9c61978d5e095810277498" exitCode=0 Jul 09 19:30:41.641654 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:41.641189 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" Jul 09 19:30:41.641654 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:41.641212 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" event={"ID":"6abe0bdd-422b-422f-a638-901fc06061e4","Type":"ContainerDied","Data":"22ebd21236f31d3d8557f4d643ada0a786bc65e62a9c61978d5e095810277498"} Jul 09 19:30:41.641654 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:41.641252 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq" event={"ID":"6abe0bdd-422b-422f-a638-901fc06061e4","Type":"ContainerDied","Data":"12f06582e848d6352c37dedb56be1ea631b0cfc317d5943783ab910b254721be"} Jul 09 19:30:41.641654 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:41.641269 2569 scope.go:117] "RemoveContainer" containerID="22ebd21236f31d3d8557f4d643ada0a786bc65e62a9c61978d5e095810277498" Jul 09 19:30:41.650502 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:41.650472 2569 scope.go:117] "RemoveContainer" containerID="22ebd21236f31d3d8557f4d643ada0a786bc65e62a9c61978d5e095810277498" Jul 09 19:30:41.650774 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:30:41.650755 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ebd21236f31d3d8557f4d643ada0a786bc65e62a9c61978d5e095810277498\": container with ID starting with 22ebd21236f31d3d8557f4d643ada0a786bc65e62a9c61978d5e095810277498 not found: ID does not exist" containerID="22ebd21236f31d3d8557f4d643ada0a786bc65e62a9c61978d5e095810277498" Jul 09 19:30:41.650842 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:41.650781 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ebd21236f31d3d8557f4d643ada0a786bc65e62a9c61978d5e095810277498"} err="failed to get container status \"22ebd21236f31d3d8557f4d643ada0a786bc65e62a9c61978d5e095810277498\": rpc error: code = NotFound desc = could not find container \"22ebd21236f31d3d8557f4d643ada0a786bc65e62a9c61978d5e095810277498\": container with ID starting with 22ebd21236f31d3d8557f4d643ada0a786bc65e62a9c61978d5e095810277498 not found: ID does not exist" Jul 09 19:30:41.663510 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:41.663482 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq"] Jul 09 19:30:41.666753 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:41.666734 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/auth-test-model-b20a51-kserve-57bcfcdd75-rzgdq"] Jul 09 19:30:41.694321 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:41.694296 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6abe0bdd-422b-422f-a638-901fc06061e4" path="/var/lib/kubelet/pods/6abe0bdd-422b-422f-a638-901fc06061e4/volumes" Jul 09 19:30:41.694684 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:30:41.694671 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da9ff940-bd8b-45e0-99f8-d0215a0f631f" path="/var/lib/kubelet/pods/da9ff940-bd8b-45e0-99f8-d0215a0f631f/volumes" Jul 09 19:31:08.049027 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.048931 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5"] Jul 09 19:31:08.049559 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.049406 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da9ff940-bd8b-45e0-99f8-d0215a0f631f" containerName="main" Jul 09 19:31:08.049559 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.049423 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="da9ff940-bd8b-45e0-99f8-d0215a0f631f" containerName="main" Jul 09 19:31:08.049559 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.049438 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6abe0bdd-422b-422f-a638-901fc06061e4" containerName="main" Jul 09 19:31:08.049559 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.049444 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6abe0bdd-422b-422f-a638-901fc06061e4" containerName="main" Jul 09 19:31:08.049559 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.049519 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="da9ff940-bd8b-45e0-99f8-d0215a0f631f" containerName="main" Jul 09 19:31:08.049559 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.049531 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6abe0bdd-422b-422f-a638-901fc06061e4" containerName="main" Jul 09 19:31:08.053928 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.053912 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.057346 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.057323 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b20a51\"/\"kube-root-ca.crt\"" Jul 09 19:31:08.057486 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.057354 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b20a51\"/\"sub-test-model-b20a51-kserve-self-signed-certs\"" Jul 09 19:31:08.057486 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.057432 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b20a51\"/\"default-dockercfg-v4pq8\"" Jul 09 19:31:08.057486 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.057357 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b20a51\"/\"openshift-service-ca.crt\"" Jul 09 19:31:08.061156 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.061134 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5"] Jul 09 19:31:08.103456 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.103428 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srdwt\" (UniqueName: \"kubernetes.io/projected/b4b1c6fa-4d00-4815-ba13-bd4077385673-kube-api-access-srdwt\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.103609 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.103462 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-dshm\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.103609 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.103489 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-model-cache\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.103609 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.103553 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-home\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.103609 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.103589 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b1c6fa-4d00-4815-ba13-bd4077385673-tls-certs\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.103746 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.103611 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-tmp-dir\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.204511 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.204466 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srdwt\" (UniqueName: \"kubernetes.io/projected/b4b1c6fa-4d00-4815-ba13-bd4077385673-kube-api-access-srdwt\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.204511 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.204511 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-dshm\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.204730 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.204532 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-model-cache\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.204730 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.204560 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-home\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.204730 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.204583 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b1c6fa-4d00-4815-ba13-bd4077385673-tls-certs\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.204730 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.204612 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-tmp-dir\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.205078 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.205049 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-home\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.205235 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.205060 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-model-cache\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.205235 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.205090 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-tmp-dir\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.206846 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.206815 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-dshm\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.207083 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.207067 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b1c6fa-4d00-4815-ba13-bd4077385673-tls-certs\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.212666 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.212642 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srdwt\" (UniqueName: \"kubernetes.io/projected/b4b1c6fa-4d00-4815-ba13-bd4077385673-kube-api-access-srdwt\") pod \"sub-test-model-b20a51-kserve-67c74bb697-hdgg5\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.364611 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.364522 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:08.492594 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.492509 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5"] Jul 09 19:31:08.494931 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:31:08.494904 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4b1c6fa_4d00_4815_ba13_bd4077385673.slice/crio-71882cb2bd65922eaed98669bdd4ea1b0553dc5ecbcf104a688143b277c1c14e WatchSource:0}: Error finding container 71882cb2bd65922eaed98669bdd4ea1b0553dc5ecbcf104a688143b277c1c14e: Status 404 returned error can't find the container with id 71882cb2bd65922eaed98669bdd4ea1b0553dc5ecbcf104a688143b277c1c14e Jul 09 19:31:08.784311 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.784271 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" event={"ID":"b4b1c6fa-4d00-4815-ba13-bd4077385673","Type":"ContainerStarted","Data":"c517cd63723281661e1a275f6a73f3b9b929693c4b0f1b4446274f33b96a4039"} Jul 09 19:31:08.784311 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.784315 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" event={"ID":"b4b1c6fa-4d00-4815-ba13-bd4077385673","Type":"ContainerStarted","Data":"71882cb2bd65922eaed98669bdd4ea1b0553dc5ecbcf104a688143b277c1c14e"} Jul 09 19:31:08.802127 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:08.802084 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" podStartSLOduration=0.802068881 podStartE2EDuration="802.068881ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:31:08.799813701 +0000 UTC m=+2253.624976737" watchObservedRunningTime="2026-07-09 19:31:08.802068881 +0000 UTC m=+2253.627231918" Jul 09 19:31:18.364995 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:18.364940 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:18.365538 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:18.365016 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:18.377916 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:18.377890 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:31:18.845924 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:31:18.845894 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:32:01.137047 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:01.137009 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5"] Jul 09 19:32:12.349368 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.349335 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv"] Jul 09 19:32:12.352673 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.352656 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.356232 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.356207 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5d8f8f\"/\"rate-test-model-5d8f8f-kserve-self-signed-certs\"" Jul 09 19:32:12.356374 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.356207 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5d8f8f\"/\"openshift-service-ca.crt\"" Jul 09 19:32:12.356374 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.356217 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5d8f8f\"/\"kube-root-ca.crt\"" Jul 09 19:32:12.356374 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.356281 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5d8f8f\"/\"default-dockercfg-crx4n\"" Jul 09 19:32:12.363067 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.363047 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv"] Jul 09 19:32:12.475213 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.475170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg9mj\" (UniqueName: \"kubernetes.io/projected/43e4edc4-0186-47d1-98ea-15927cd33acc-kube-api-access-qg9mj\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.475392 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.475223 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-model-cache\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.475392 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.475267 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-home\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.475392 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.475282 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-dshm\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.475505 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.475411 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e4edc4-0186-47d1-98ea-15927cd33acc-tls-certs\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.475505 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.475439 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-tmp-dir\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.575852 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.575816 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e4edc4-0186-47d1-98ea-15927cd33acc-tls-certs\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.575852 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.575851 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-tmp-dir\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.576122 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.575878 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg9mj\" (UniqueName: \"kubernetes.io/projected/43e4edc4-0186-47d1-98ea-15927cd33acc-kube-api-access-qg9mj\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.576122 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.575917 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-model-cache\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.576122 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.575970 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-home\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.576122 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.575993 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-dshm\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.576337 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.576310 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-tmp-dir\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.576337 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.576323 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-home\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.576442 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.576375 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-model-cache\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.578267 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.578238 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-dshm\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.578438 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.578421 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e4edc4-0186-47d1-98ea-15927cd33acc-tls-certs\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.585299 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.585273 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg9mj\" (UniqueName: \"kubernetes.io/projected/43e4edc4-0186-47d1-98ea-15927cd33acc-kube-api-access-qg9mj\") pod \"rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.662811 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.662714 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:12.792598 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:12.792571 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv"] Jul 09 19:32:12.794758 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:32:12.794730 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43e4edc4_0186_47d1_98ea_15927cd33acc.slice/crio-9a785a474a0f3c972168b28f299e56d1412a08308362e4792ed36ce45c116b00 WatchSource:0}: Error finding container 9a785a474a0f3c972168b28f299e56d1412a08308362e4792ed36ce45c116b00: Status 404 returned error can't find the container with id 9a785a474a0f3c972168b28f299e56d1412a08308362e4792ed36ce45c116b00 Jul 09 19:32:13.106457 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:13.106413 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" event={"ID":"43e4edc4-0186-47d1-98ea-15927cd33acc","Type":"ContainerStarted","Data":"142d1a868498754580e7cfaa468d79ce5df6e70f91b75782feb9ac4f023bbc8c"} Jul 09 19:32:13.106457 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:13.106461 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" event={"ID":"43e4edc4-0186-47d1-98ea-15927cd33acc","Type":"ContainerStarted","Data":"9a785a474a0f3c972168b28f299e56d1412a08308362e4792ed36ce45c116b00"} Jul 09 19:32:13.124041 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:13.123994 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" podStartSLOduration=1.123975401 podStartE2EDuration="1.123975401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:32:13.12247946 +0000 UTC m=+2317.947642507" watchObservedRunningTime="2026-07-09 19:32:13.123975401 +0000 UTC m=+2317.949138438" Jul 09 19:32:16.156247 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.156210 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" podUID="b4b1c6fa-4d00-4815-ba13-bd4077385673" containerName="main" containerID="cri-o://c517cd63723281661e1a275f6a73f3b9b929693c4b0f1b4446274f33b96a4039" gracePeriod=45 Jul 09 19:32:16.401680 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.401654 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:32:16.513154 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.513123 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srdwt\" (UniqueName: \"kubernetes.io/projected/b4b1c6fa-4d00-4815-ba13-bd4077385673-kube-api-access-srdwt\") pod \"b4b1c6fa-4d00-4815-ba13-bd4077385673\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " Jul 09 19:32:16.513321 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.513167 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-model-cache\") pod \"b4b1c6fa-4d00-4815-ba13-bd4077385673\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " Jul 09 19:32:16.513321 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.513227 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-dshm\") pod \"b4b1c6fa-4d00-4815-ba13-bd4077385673\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " Jul 09 19:32:16.513321 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.513251 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-tmp-dir\") pod \"b4b1c6fa-4d00-4815-ba13-bd4077385673\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " Jul 09 19:32:16.513321 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.513275 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-home\") pod \"b4b1c6fa-4d00-4815-ba13-bd4077385673\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " Jul 09 19:32:16.513321 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.513296 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b1c6fa-4d00-4815-ba13-bd4077385673-tls-certs\") pod \"b4b1c6fa-4d00-4815-ba13-bd4077385673\" (UID: \"b4b1c6fa-4d00-4815-ba13-bd4077385673\") " Jul 09 19:32:16.513647 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.513485 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4b1c6fa-4d00-4815-ba13-bd4077385673" (UID: "b4b1c6fa-4d00-4815-ba13-bd4077385673"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:32:16.513647 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.513497 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4b1c6fa-4d00-4815-ba13-bd4077385673" (UID: "b4b1c6fa-4d00-4815-ba13-bd4077385673"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:32:16.513647 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.513547 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-home" (OuterVolumeSpecName: "home") pod "b4b1c6fa-4d00-4815-ba13-bd4077385673" (UID: "b4b1c6fa-4d00-4815-ba13-bd4077385673"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:32:16.513647 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.513633 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-model-cache\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:32:16.513794 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.513650 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-tmp-dir\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:32:16.513794 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.513665 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-home\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:32:16.515365 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.515339 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b1c6fa-4d00-4815-ba13-bd4077385673-kube-api-access-srdwt" (OuterVolumeSpecName: "kube-api-access-srdwt") pod "b4b1c6fa-4d00-4815-ba13-bd4077385673" (UID: "b4b1c6fa-4d00-4815-ba13-bd4077385673"). InnerVolumeSpecName "kube-api-access-srdwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:32:16.515365 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.515356 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4b1c6fa-4d00-4815-ba13-bd4077385673-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4b1c6fa-4d00-4815-ba13-bd4077385673" (UID: "b4b1c6fa-4d00-4815-ba13-bd4077385673"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:32:16.515501 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.515371 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-dshm" (OuterVolumeSpecName: "dshm") pod "b4b1c6fa-4d00-4815-ba13-bd4077385673" (UID: "b4b1c6fa-4d00-4815-ba13-bd4077385673"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:32:16.614117 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.614082 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4b1c6fa-4d00-4815-ba13-bd4077385673-dshm\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:32:16.614117 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.614110 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b1c6fa-4d00-4815-ba13-bd4077385673-tls-certs\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:32:16.614117 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:16.614121 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srdwt\" (UniqueName: \"kubernetes.io/projected/b4b1c6fa-4d00-4815-ba13-bd4077385673-kube-api-access-srdwt\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:32:17.129515 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:17.129480 2569 generic.go:356] "Generic (PLEG): container finished" podID="b4b1c6fa-4d00-4815-ba13-bd4077385673" containerID="c517cd63723281661e1a275f6a73f3b9b929693c4b0f1b4446274f33b96a4039" exitCode=0 Jul 09 19:32:17.129730 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:17.129522 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" Jul 09 19:32:17.129730 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:17.129575 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" event={"ID":"b4b1c6fa-4d00-4815-ba13-bd4077385673","Type":"ContainerDied","Data":"c517cd63723281661e1a275f6a73f3b9b929693c4b0f1b4446274f33b96a4039"} Jul 09 19:32:17.129730 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:17.129623 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5" event={"ID":"b4b1c6fa-4d00-4815-ba13-bd4077385673","Type":"ContainerDied","Data":"71882cb2bd65922eaed98669bdd4ea1b0553dc5ecbcf104a688143b277c1c14e"} Jul 09 19:32:17.129730 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:17.129645 2569 scope.go:117] "RemoveContainer" containerID="c517cd63723281661e1a275f6a73f3b9b929693c4b0f1b4446274f33b96a4039" Jul 09 19:32:17.141835 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:17.141815 2569 scope.go:117] "RemoveContainer" containerID="c517cd63723281661e1a275f6a73f3b9b929693c4b0f1b4446274f33b96a4039" Jul 09 19:32:17.142176 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:32:17.142156 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c517cd63723281661e1a275f6a73f3b9b929693c4b0f1b4446274f33b96a4039\": container with ID starting with c517cd63723281661e1a275f6a73f3b9b929693c4b0f1b4446274f33b96a4039 not found: ID does not exist" containerID="c517cd63723281661e1a275f6a73f3b9b929693c4b0f1b4446274f33b96a4039" Jul 09 19:32:17.142261 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:17.142184 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c517cd63723281661e1a275f6a73f3b9b929693c4b0f1b4446274f33b96a4039"} err="failed to get container status \"c517cd63723281661e1a275f6a73f3b9b929693c4b0f1b4446274f33b96a4039\": rpc error: code = NotFound desc = could not find container \"c517cd63723281661e1a275f6a73f3b9b929693c4b0f1b4446274f33b96a4039\": container with ID starting with c517cd63723281661e1a275f6a73f3b9b929693c4b0f1b4446274f33b96a4039 not found: ID does not exist" Jul 09 19:32:17.152012 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:17.151990 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5"] Jul 09 19:32:17.156357 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:17.156338 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/sub-test-model-b20a51-kserve-67c74bb697-hdgg5"] Jul 09 19:32:17.695062 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:17.695022 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4b1c6fa-4d00-4815-ba13-bd4077385673" path="/var/lib/kubelet/pods/b4b1c6fa-4d00-4815-ba13-bd4077385673/volumes" Jul 09 19:32:22.663854 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:22.663816 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:22.664276 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:22.664115 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:22.676361 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:22.676340 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:23.172027 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:23.171996 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:32:44.970497 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:44.970464 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq"] Jul 09 19:32:44.970932 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:44.970799 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b1c6fa-4d00-4815-ba13-bd4077385673" containerName="main" Jul 09 19:32:44.970932 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:44.970809 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b1c6fa-4d00-4815-ba13-bd4077385673" containerName="main" Jul 09 19:32:44.970932 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:44.970871 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4b1c6fa-4d00-4815-ba13-bd4077385673" containerName="main" Jul 09 19:32:44.973920 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:44.973903 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:44.976837 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:44.976816 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b20a51\"/\"kube-root-ca.crt\"" Jul 09 19:32:44.977627 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:44.977600 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b20a51\"/\"default-dockercfg-v4pq8\"" Jul 09 19:32:44.977750 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:44.977635 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b20a51\"/\"openshift-service-ca.crt\"" Jul 09 19:32:44.977750 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:44.977721 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b20a51\"/\"rate-test-model-b20a51-kserve-self-signed-certs\"" Jul 09 19:32:44.986009 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:44.985985 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq"] Jul 09 19:32:45.062707 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.062659 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-tmp-dir\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.062707 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.062713 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-dshm\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.062923 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.062794 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-home\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.062923 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.062816 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-model-cache\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.062923 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.062840 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46b74bc-0e75-4001-ba18-da0df10e1d72-tls-certs\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.062923 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.062861 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdvv9\" (UniqueName: \"kubernetes.io/projected/f46b74bc-0e75-4001-ba18-da0df10e1d72-kube-api-access-qdvv9\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.164303 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.164255 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-home\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.164303 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.164310 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-model-cache\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.164580 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.164430 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46b74bc-0e75-4001-ba18-da0df10e1d72-tls-certs\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.164580 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.164471 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdvv9\" (UniqueName: \"kubernetes.io/projected/f46b74bc-0e75-4001-ba18-da0df10e1d72-kube-api-access-qdvv9\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.164580 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.164507 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-tmp-dir\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.164580 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.164539 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-dshm\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.164790 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.164652 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-model-cache\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.164790 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.164732 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-home\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.164867 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.164827 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-tmp-dir\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.166682 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.166655 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-dshm\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.167099 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.167077 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46b74bc-0e75-4001-ba18-da0df10e1d72-tls-certs\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.175475 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.175447 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdvv9\" (UniqueName: \"kubernetes.io/projected/f46b74bc-0e75-4001-ba18-da0df10e1d72-kube-api-access-qdvv9\") pod \"rate-test-model-b20a51-kserve-67689fd6b7-tl5jq\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.284003 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.283902 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:45.427646 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:32:45.427613 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf46b74bc_0e75_4001_ba18_da0df10e1d72.slice/crio-81c16b48fb866a9889183eb02c7332dee1519aab62bd182d751d14b56d5197d4 WatchSource:0}: Error finding container 81c16b48fb866a9889183eb02c7332dee1519aab62bd182d751d14b56d5197d4: Status 404 returned error can't find the container with id 81c16b48fb866a9889183eb02c7332dee1519aab62bd182d751d14b56d5197d4 Jul 09 19:32:45.428525 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:45.428506 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq"] Jul 09 19:32:46.275868 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:46.275830 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" event={"ID":"f46b74bc-0e75-4001-ba18-da0df10e1d72","Type":"ContainerStarted","Data":"c4df52ee1f9b18c0c988b6c2f6248c847914c47b6035e124bb4c695059fc5a0b"} Jul 09 19:32:46.276260 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:46.275876 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" event={"ID":"f46b74bc-0e75-4001-ba18-da0df10e1d72","Type":"ContainerStarted","Data":"81c16b48fb866a9889183eb02c7332dee1519aab62bd182d751d14b56d5197d4"} Jul 09 19:32:46.296657 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:46.296609 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" podStartSLOduration=2.296591741 podStartE2EDuration="2.296591741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:32:46.294868793 +0000 UTC m=+2351.120031831" watchObservedRunningTime="2026-07-09 19:32:46.296591741 +0000 UTC m=+2351.121754783" Jul 09 19:32:55.284721 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:55.284687 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:55.285166 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:55.284733 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:55.297448 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:55.297423 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:32:55.332246 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:32:55.332223 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:33:32.654820 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:32.654785 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv"] Jul 09 19:33:33.340438 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:33.340408 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq"] Jul 09 19:33:35.811000 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:35.810976 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 19:33:35.819641 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:35.819622 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsr6w_348e3292-41e0-44d6-b59b-81372e7d0b81/ovn-acl-logging/0.log" Jul 09 19:33:47.675349 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:47.675286 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" podUID="43e4edc4-0186-47d1-98ea-15927cd33acc" containerName="main" containerID="cri-o://142d1a868498754580e7cfaa468d79ce5df6e70f91b75782feb9ac4f023bbc8c" gracePeriod=45 Jul 09 19:33:47.921389 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:47.921366 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:33:48.022691 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.022656 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-tmp-dir\") pod \"43e4edc4-0186-47d1-98ea-15927cd33acc\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " Jul 09 19:33:48.022898 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.022761 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-model-cache\") pod \"43e4edc4-0186-47d1-98ea-15927cd33acc\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " Jul 09 19:33:48.022898 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.022804 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg9mj\" (UniqueName: \"kubernetes.io/projected/43e4edc4-0186-47d1-98ea-15927cd33acc-kube-api-access-qg9mj\") pod \"43e4edc4-0186-47d1-98ea-15927cd33acc\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " Jul 09 19:33:48.022898 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.022851 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e4edc4-0186-47d1-98ea-15927cd33acc-tls-certs\") pod \"43e4edc4-0186-47d1-98ea-15927cd33acc\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " Jul 09 19:33:48.023112 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.022933 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-home\") pod \"43e4edc4-0186-47d1-98ea-15927cd33acc\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " Jul 09 19:33:48.023112 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.022986 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-dshm\") pod \"43e4edc4-0186-47d1-98ea-15927cd33acc\" (UID: \"43e4edc4-0186-47d1-98ea-15927cd33acc\") " Jul 09 19:33:48.023112 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.023020 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43e4edc4-0186-47d1-98ea-15927cd33acc" (UID: "43e4edc4-0186-47d1-98ea-15927cd33acc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:33:48.023112 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.023030 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-model-cache" (OuterVolumeSpecName: "model-cache") pod "43e4edc4-0186-47d1-98ea-15927cd33acc" (UID: "43e4edc4-0186-47d1-98ea-15927cd33acc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:33:48.023302 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.023237 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-home" (OuterVolumeSpecName: "home") pod "43e4edc4-0186-47d1-98ea-15927cd33acc" (UID: "43e4edc4-0186-47d1-98ea-15927cd33acc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:33:48.023302 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.023248 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-tmp-dir\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:33:48.023302 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.023286 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-model-cache\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:33:48.025048 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.025024 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-dshm" (OuterVolumeSpecName: "dshm") pod "43e4edc4-0186-47d1-98ea-15927cd33acc" (UID: "43e4edc4-0186-47d1-98ea-15927cd33acc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:33:48.025137 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.025027 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e4edc4-0186-47d1-98ea-15927cd33acc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43e4edc4-0186-47d1-98ea-15927cd33acc" (UID: "43e4edc4-0186-47d1-98ea-15927cd33acc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:33:48.025137 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.025110 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e4edc4-0186-47d1-98ea-15927cd33acc-kube-api-access-qg9mj" (OuterVolumeSpecName: "kube-api-access-qg9mj") pod "43e4edc4-0186-47d1-98ea-15927cd33acc" (UID: "43e4edc4-0186-47d1-98ea-15927cd33acc"). InnerVolumeSpecName "kube-api-access-qg9mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:33:48.124185 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.124150 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg9mj\" (UniqueName: \"kubernetes.io/projected/43e4edc4-0186-47d1-98ea-15927cd33acc-kube-api-access-qg9mj\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:33:48.124185 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.124179 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e4edc4-0186-47d1-98ea-15927cd33acc-tls-certs\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:33:48.124185 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.124189 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-home\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:33:48.124419 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.124197 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43e4edc4-0186-47d1-98ea-15927cd33acc-dshm\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:33:48.359930 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.359840 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" podUID="f46b74bc-0e75-4001-ba18-da0df10e1d72" containerName="main" containerID="cri-o://c4df52ee1f9b18c0c988b6c2f6248c847914c47b6035e124bb4c695059fc5a0b" gracePeriod=45 Jul 09 19:33:48.588885 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.588852 2569 generic.go:356] "Generic (PLEG): container finished" podID="f46b74bc-0e75-4001-ba18-da0df10e1d72" containerID="c4df52ee1f9b18c0c988b6c2f6248c847914c47b6035e124bb4c695059fc5a0b" exitCode=0 Jul 09 19:33:48.589080 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.588920 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" event={"ID":"f46b74bc-0e75-4001-ba18-da0df10e1d72","Type":"ContainerDied","Data":"c4df52ee1f9b18c0c988b6c2f6248c847914c47b6035e124bb4c695059fc5a0b"} Jul 09 19:33:48.590302 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.590268 2569 generic.go:356] "Generic (PLEG): container finished" podID="43e4edc4-0186-47d1-98ea-15927cd33acc" containerID="142d1a868498754580e7cfaa468d79ce5df6e70f91b75782feb9ac4f023bbc8c" exitCode=0 Jul 09 19:33:48.590448 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.590313 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" Jul 09 19:33:48.590448 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.590319 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" event={"ID":"43e4edc4-0186-47d1-98ea-15927cd33acc","Type":"ContainerDied","Data":"142d1a868498754580e7cfaa468d79ce5df6e70f91b75782feb9ac4f023bbc8c"} Jul 09 19:33:48.590448 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.590357 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv" event={"ID":"43e4edc4-0186-47d1-98ea-15927cd33acc","Type":"ContainerDied","Data":"9a785a474a0f3c972168b28f299e56d1412a08308362e4792ed36ce45c116b00"} Jul 09 19:33:48.590448 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.590375 2569 scope.go:117] "RemoveContainer" containerID="142d1a868498754580e7cfaa468d79ce5df6e70f91b75782feb9ac4f023bbc8c" Jul 09 19:33:48.599377 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.599361 2569 scope.go:117] "RemoveContainer" containerID="142d1a868498754580e7cfaa468d79ce5df6e70f91b75782feb9ac4f023bbc8c" Jul 09 19:33:48.599610 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:33:48.599592 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"142d1a868498754580e7cfaa468d79ce5df6e70f91b75782feb9ac4f023bbc8c\": container with ID starting with 142d1a868498754580e7cfaa468d79ce5df6e70f91b75782feb9ac4f023bbc8c not found: ID does not exist" containerID="142d1a868498754580e7cfaa468d79ce5df6e70f91b75782feb9ac4f023bbc8c" Jul 09 19:33:48.599666 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.599620 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"142d1a868498754580e7cfaa468d79ce5df6e70f91b75782feb9ac4f023bbc8c"} err="failed to get container status \"142d1a868498754580e7cfaa468d79ce5df6e70f91b75782feb9ac4f023bbc8c\": rpc error: code = NotFound desc = could not find container \"142d1a868498754580e7cfaa468d79ce5df6e70f91b75782feb9ac4f023bbc8c\": container with ID starting with 142d1a868498754580e7cfaa468d79ce5df6e70f91b75782feb9ac4f023bbc8c not found: ID does not exist" Jul 09 19:33:48.600554 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.600535 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:33:48.612519 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.612455 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv"] Jul 09 19:33:48.615795 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.615774 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/rate-test-model-5d8f8f-kserve-67cc948bc-sd6cv"] Jul 09 19:33:48.730151 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.730121 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-home\") pod \"f46b74bc-0e75-4001-ba18-da0df10e1d72\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " Jul 09 19:33:48.730559 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.730160 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdvv9\" (UniqueName: \"kubernetes.io/projected/f46b74bc-0e75-4001-ba18-da0df10e1d72-kube-api-access-qdvv9\") pod \"f46b74bc-0e75-4001-ba18-da0df10e1d72\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " Jul 09 19:33:48.730559 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.730229 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46b74bc-0e75-4001-ba18-da0df10e1d72-tls-certs\") pod \"f46b74bc-0e75-4001-ba18-da0df10e1d72\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " Jul 09 19:33:48.730559 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.730306 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-tmp-dir\") pod \"f46b74bc-0e75-4001-ba18-da0df10e1d72\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " Jul 09 19:33:48.730559 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.730359 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-model-cache\") pod \"f46b74bc-0e75-4001-ba18-da0df10e1d72\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " Jul 09 19:33:48.730559 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.730425 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-dshm\") pod \"f46b74bc-0e75-4001-ba18-da0df10e1d72\" (UID: \"f46b74bc-0e75-4001-ba18-da0df10e1d72\") " Jul 09 19:33:48.730559 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.730444 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-home" (OuterVolumeSpecName: "home") pod "f46b74bc-0e75-4001-ba18-da0df10e1d72" (UID: "f46b74bc-0e75-4001-ba18-da0df10e1d72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:33:48.730559 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.730492 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f46b74bc-0e75-4001-ba18-da0df10e1d72" (UID: "f46b74bc-0e75-4001-ba18-da0df10e1d72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:33:48.730889 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.730648 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-model-cache" (OuterVolumeSpecName: "model-cache") pod "f46b74bc-0e75-4001-ba18-da0df10e1d72" (UID: "f46b74bc-0e75-4001-ba18-da0df10e1d72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:33:48.730889 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.730772 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-tmp-dir\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:33:48.730889 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.730791 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-model-cache\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:33:48.730889 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.730805 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-home\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:33:48.732395 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.732371 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-dshm" (OuterVolumeSpecName: "dshm") pod "f46b74bc-0e75-4001-ba18-da0df10e1d72" (UID: "f46b74bc-0e75-4001-ba18-da0df10e1d72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:33:48.732395 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.732391 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46b74bc-0e75-4001-ba18-da0df10e1d72-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f46b74bc-0e75-4001-ba18-da0df10e1d72" (UID: "f46b74bc-0e75-4001-ba18-da0df10e1d72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:33:48.732535 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.732442 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46b74bc-0e75-4001-ba18-da0df10e1d72-kube-api-access-qdvv9" (OuterVolumeSpecName: "kube-api-access-qdvv9") pod "f46b74bc-0e75-4001-ba18-da0df10e1d72" (UID: "f46b74bc-0e75-4001-ba18-da0df10e1d72"). InnerVolumeSpecName "kube-api-access-qdvv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:33:48.831452 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.831419 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f46b74bc-0e75-4001-ba18-da0df10e1d72-dshm\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:33:48.831452 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.831456 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdvv9\" (UniqueName: \"kubernetes.io/projected/f46b74bc-0e75-4001-ba18-da0df10e1d72-kube-api-access-qdvv9\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:33:48.831657 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:48.831471 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46b74bc-0e75-4001-ba18-da0df10e1d72-tls-certs\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:33:49.376616 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.376583 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7482-140307192413040"] Jul 09 19:33:49.377033 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.377016 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e4edc4-0186-47d1-98ea-15927cd33acc" containerName="main" Jul 09 19:33:49.377033 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.377035 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e4edc4-0186-47d1-98ea-15927cd33acc" containerName="main" Jul 09 19:33:49.377203 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.377047 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46b74bc-0e75-4001-ba18-da0df10e1d72" containerName="main" Jul 09 19:33:49.377203 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.377055 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46b74bc-0e75-4001-ba18-da0df10e1d72" containerName="main" Jul 09 19:33:49.377203 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.377141 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e4edc4-0186-47d1-98ea-15927cd33acc" containerName="main" Jul 09 19:33:49.377203 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.377153 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f46b74bc-0e75-4001-ba18-da0df10e1d72" containerName="main" Jul 09 19:33:49.381428 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.381402 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192413040" Jul 09 19:33:49.383604 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.383585 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-7trn4\"" Jul 09 19:33:49.384095 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.384070 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7482-140307192413040"] Jul 09 19:33:49.538578 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.538537 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66nh8\" (UniqueName: \"kubernetes.io/projected/0094950f-5cd4-48a9-bb4f-d05967cf90cf-kube-api-access-66nh8\") pod \"test-curl-7482-140307192413040\" (UID: \"0094950f-5cd4-48a9-bb4f-d05967cf90cf\") " pod="openshift-ingress/test-curl-7482-140307192413040" Jul 09 19:33:49.596740 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.596712 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" Jul 09 19:33:49.596934 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.596712 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq" event={"ID":"f46b74bc-0e75-4001-ba18-da0df10e1d72","Type":"ContainerDied","Data":"81c16b48fb866a9889183eb02c7332dee1519aab62bd182d751d14b56d5197d4"} Jul 09 19:33:49.596934 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.596829 2569 scope.go:117] "RemoveContainer" containerID="c4df52ee1f9b18c0c988b6c2f6248c847914c47b6035e124bb4c695059fc5a0b" Jul 09 19:33:49.620798 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.620761 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq"] Jul 09 19:33:49.626481 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.626456 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b20a51/rate-test-model-b20a51-kserve-67689fd6b7-tl5jq"] Jul 09 19:33:49.639794 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.639741 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66nh8\" (UniqueName: \"kubernetes.io/projected/0094950f-5cd4-48a9-bb4f-d05967cf90cf-kube-api-access-66nh8\") pod \"test-curl-7482-140307192413040\" (UID: \"0094950f-5cd4-48a9-bb4f-d05967cf90cf\") " pod="openshift-ingress/test-curl-7482-140307192413040" Jul 09 19:33:49.647451 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.647429 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66nh8\" (UniqueName: \"kubernetes.io/projected/0094950f-5cd4-48a9-bb4f-d05967cf90cf-kube-api-access-66nh8\") pod \"test-curl-7482-140307192413040\" (UID: \"0094950f-5cd4-48a9-bb4f-d05967cf90cf\") " pod="openshift-ingress/test-curl-7482-140307192413040" Jul 09 19:33:49.691231 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.691200 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192413040" Jul 09 19:33:49.694591 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.694569 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43e4edc4-0186-47d1-98ea-15927cd33acc" path="/var/lib/kubelet/pods/43e4edc4-0186-47d1-98ea-15927cd33acc/volumes" Jul 09 19:33:49.694979 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.694962 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f46b74bc-0e75-4001-ba18-da0df10e1d72" path="/var/lib/kubelet/pods/f46b74bc-0e75-4001-ba18-da0df10e1d72/volumes" Jul 09 19:33:49.822227 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.822202 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7482-140307192413040"] Jul 09 19:33:49.823494 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:33:49.823463 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0094950f_5cd4_48a9_bb4f_d05967cf90cf.slice/crio-92c15bf3a87a23d51f12118f51f4188412b6239a00723f491426109d8a63965e WatchSource:0}: Error finding container 92c15bf3a87a23d51f12118f51f4188412b6239a00723f491426109d8a63965e: Status 404 returned error can't find the container with id 92c15bf3a87a23d51f12118f51f4188412b6239a00723f491426109d8a63965e Jul 09 19:33:49.824669 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:49.824652 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 19:33:50.605332 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:50.605302 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7482-140307192413040" event={"ID":"0094950f-5cd4-48a9-bb4f-d05967cf90cf","Type":"ContainerStarted","Data":"92c15bf3a87a23d51f12118f51f4188412b6239a00723f491426109d8a63965e"} Jul 09 19:33:51.611446 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:51.611408 2569 generic.go:356] "Generic (PLEG): container finished" podID="0094950f-5cd4-48a9-bb4f-d05967cf90cf" containerID="67af01c50353b6110000d4ed631a2c667e64730fa7b733376b48b5dc57a73be8" exitCode=0 Jul 09 19:33:51.611853 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:51.611498 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7482-140307192413040" event={"ID":"0094950f-5cd4-48a9-bb4f-d05967cf90cf","Type":"ContainerDied","Data":"67af01c50353b6110000d4ed631a2c667e64730fa7b733376b48b5dc57a73be8"} Jul 09 19:33:52.752895 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:52.752872 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192413040" Jul 09 19:33:52.787430 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:52.787402 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7482-140307192413040_0094950f-5cd4-48a9-bb4f-d05967cf90cf/test-curl-7482-140307192413040/0.log" Jul 09 19:33:52.814514 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:52.814486 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7482-140307192413040"] Jul 09 19:33:52.818590 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:52.818568 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7482-140307192413040"] Jul 09 19:33:52.867320 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:52.867241 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66nh8\" (UniqueName: \"kubernetes.io/projected/0094950f-5cd4-48a9-bb4f-d05967cf90cf-kube-api-access-66nh8\") pod \"0094950f-5cd4-48a9-bb4f-d05967cf90cf\" (UID: \"0094950f-5cd4-48a9-bb4f-d05967cf90cf\") " Jul 09 19:33:52.869415 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:52.869387 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0094950f-5cd4-48a9-bb4f-d05967cf90cf-kube-api-access-66nh8" (OuterVolumeSpecName: "kube-api-access-66nh8") pod "0094950f-5cd4-48a9-bb4f-d05967cf90cf" (UID: "0094950f-5cd4-48a9-bb4f-d05967cf90cf"). InnerVolumeSpecName "kube-api-access-66nh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:33:52.967962 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:52.967922 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66nh8\" (UniqueName: \"kubernetes.io/projected/0094950f-5cd4-48a9-bb4f-d05967cf90cf-kube-api-access-66nh8\") on node \"ip-10-0-133-185.ec2.internal\" DevicePath \"\"" Jul 09 19:33:53.622983 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:53.622939 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192413040" Jul 09 19:33:53.623193 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:53.622994 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92c15bf3a87a23d51f12118f51f4188412b6239a00723f491426109d8a63965e" Jul 09 19:33:53.632550 ip-10-0-133-185 kubenswrapper[2569]: E0709 19:33:53.632524 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7482-140307192413040\" is forbidden: User \"system:node:ip-10-0-133-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-185.ec2.internal' and this object" podUID="0094950f-5cd4-48a9-bb4f-d05967cf90cf" pod="openshift-ingress/test-curl-7482-140307192413040" Jul 09 19:33:53.694468 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:33:53.694440 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0094950f-5cd4-48a9-bb4f-d05967cf90cf" path="/var/lib/kubelet/pods/0094950f-5cd4-48a9-bb4f-d05967cf90cf/volumes" Jul 09 19:34:07.756147 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:07.756076 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-867fd7fc68-r5r5s_dcab9734-a390-4768-8dc6-13ed97aa6b8d/authorino/0.log" Jul 09 19:34:12.833511 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:12.833479 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-698jk_20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4/manager/0.log" Jul 09 19:34:13.585995 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:13.585966 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cd98cb899-p9bm8_11642f9a-aa84-47c8-b35d-1f4aa0397211/copy-manifests/0.log" Jul 09 19:34:13.593530 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:13.593503 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cd98cb899-p9bm8_11642f9a-aa84-47c8-b35d-1f4aa0397211/manager/0.log" Jul 09 19:34:14.860965 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:14.860911 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh_6bd73eca-bfc6-4d0a-950b-0875ff85f40b/pull/0.log" Jul 09 19:34:14.869781 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:14.869759 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh_6bd73eca-bfc6-4d0a-950b-0875ff85f40b/extract/0.log" Jul 09 19:34:14.875778 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:14.875757 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh_6bd73eca-bfc6-4d0a-950b-0875ff85f40b/util/0.log" Jul 09 19:34:14.986388 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:14.986360 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k_bb6716eb-53f8-48f9-a8c6-1484cd809ed1/util/0.log" Jul 09 19:34:14.993464 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:14.993436 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k_bb6716eb-53f8-48f9-a8c6-1484cd809ed1/pull/0.log" Jul 09 19:34:15.000528 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:15.000507 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k_bb6716eb-53f8-48f9-a8c6-1484cd809ed1/extract/0.log" Jul 09 19:34:15.114831 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:15.114742 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv_7aa80757-737d-4805-981a-e102986f8479/util/0.log" Jul 09 19:34:15.122309 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:15.122281 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv_7aa80757-737d-4805-981a-e102986f8479/pull/0.log" Jul 09 19:34:15.129696 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:15.129672 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv_7aa80757-737d-4805-981a-e102986f8479/extract/0.log" Jul 09 19:34:15.254509 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:15.254481 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9_03a598a3-74c3-4360-946b-c8fbf548dcf6/extract/0.log" Jul 09 19:34:15.268465 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:15.268443 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9_03a598a3-74c3-4360-946b-c8fbf548dcf6/util/0.log" Jul 09 19:34:15.285313 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:15.285286 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9_03a598a3-74c3-4360-946b-c8fbf548dcf6/pull/0.log" Jul 09 19:34:15.427029 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:15.426927 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-867fd7fc68-r5r5s_dcab9734-a390-4768-8dc6-13ed97aa6b8d/authorino/0.log" Jul 09 19:34:15.785563 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:15.785535 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-kvflz_21b026d8-5239-4098-b1e7-3cdcf6f07bc1/kuadrant-console-plugin/0.log" Jul 09 19:34:16.871813 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:16.871786 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58cdb447-4kr8q_73e08385-5634-4158-928d-7c840cc3216b/kube-auth-proxy/0.log" Jul 09 19:34:17.242372 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:17.242343 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-nxrmk_c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52/payload-pre-processing/0.log" Jul 09 19:34:17.836337 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:17.836309 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-rkwpc_f6ea3f9d-42a8-4a41-8c01-9b34d5d2a712/main/0.log" Jul 09 19:34:18.068176 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:18.068126 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-b5cd5_503b4318-56c2-4696-8b5f-203b229ba825/main/0.log" Jul 09 19:34:18.299178 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:18.299149 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-h476q_c602f2e4-e262-4830-b826-58070d1e5caf/main/0.log" Jul 09 19:34:18.408418 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:18.408384 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-w6g4h_3be36501-e4c2-4d76-9b42-fa7578ab7932/main/0.log" Jul 09 19:34:25.695477 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:25.695440 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kss86_8de73e1e-1dc7-4c83-98fd-ac2697ca4c15/global-pull-secret-syncer/0.log" Jul 09 19:34:25.841982 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:25.841935 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lmbqh_dd7d71e8-6272-469f-8569-7d97ac86c975/konnectivity-agent/0.log" Jul 09 19:34:25.986267 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:25.986233 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-185.ec2.internal_58be071d26cafa55b0bf4c584a43070e/haproxy/0.log" Jul 09 19:34:31.230923 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:31.230889 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh_6bd73eca-bfc6-4d0a-950b-0875ff85f40b/extract/0.log" Jul 09 19:34:31.250075 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:31.250053 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh_6bd73eca-bfc6-4d0a-950b-0875ff85f40b/util/0.log" Jul 09 19:34:31.271338 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:31.271312 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599vldh_6bd73eca-bfc6-4d0a-950b-0875ff85f40b/pull/0.log" Jul 09 19:34:31.299697 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:31.299670 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k_bb6716eb-53f8-48f9-a8c6-1484cd809ed1/extract/0.log" Jul 09 19:34:31.322686 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:31.322666 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k_bb6716eb-53f8-48f9-a8c6-1484cd809ed1/util/0.log" Jul 09 19:34:31.345677 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:31.345653 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vq86k_bb6716eb-53f8-48f9-a8c6-1484cd809ed1/pull/0.log" Jul 09 19:34:31.383152 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:31.383128 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv_7aa80757-737d-4805-981a-e102986f8479/extract/0.log" Jul 09 19:34:31.409110 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:31.409087 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv_7aa80757-737d-4805-981a-e102986f8479/util/0.log" Jul 09 19:34:31.438228 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:31.438203 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73dnlvv_7aa80757-737d-4805-981a-e102986f8479/pull/0.log" Jul 09 19:34:31.476567 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:31.476544 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9_03a598a3-74c3-4360-946b-c8fbf548dcf6/extract/0.log" Jul 09 19:34:31.512069 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:31.512011 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9_03a598a3-74c3-4360-946b-c8fbf548dcf6/util/0.log" Jul 09 19:34:31.536933 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:31.536909 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mqfh9_03a598a3-74c3-4360-946b-c8fbf548dcf6/pull/0.log" Jul 09 19:34:32.041119 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:32.041093 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-867fd7fc68-r5r5s_dcab9734-a390-4768-8dc6-13ed97aa6b8d/authorino/0.log" Jul 09 19:34:32.129609 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:32.129585 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-kvflz_21b026d8-5239-4098-b1e7-3cdcf6f07bc1/kuadrant-console-plugin/0.log" Jul 09 19:34:34.019829 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.019795 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59b649d0-cbd9-4828-a8b3-13fa14188249/alertmanager/0.log" Jul 09 19:34:34.042370 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.042345 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59b649d0-cbd9-4828-a8b3-13fa14188249/config-reloader/0.log" Jul 09 19:34:34.063895 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.063867 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59b649d0-cbd9-4828-a8b3-13fa14188249/kube-rbac-proxy-web/0.log" Jul 09 19:34:34.092150 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.092125 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59b649d0-cbd9-4828-a8b3-13fa14188249/kube-rbac-proxy/0.log" Jul 09 19:34:34.114796 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.114773 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59b649d0-cbd9-4828-a8b3-13fa14188249/kube-rbac-proxy-metric/0.log" Jul 09 19:34:34.136481 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.136461 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59b649d0-cbd9-4828-a8b3-13fa14188249/prom-label-proxy/0.log" Jul 09 19:34:34.159173 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.159153 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59b649d0-cbd9-4828-a8b3-13fa14188249/init-config-reloader/0.log" Jul 09 19:34:34.234009 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.233985 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-h27h2_74defd5b-37e2-48c7-9b3c-849720978d12/kube-state-metrics/0.log" Jul 09 19:34:34.257611 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.257592 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-h27h2_74defd5b-37e2-48c7-9b3c-849720978d12/kube-rbac-proxy-main/0.log" Jul 09 19:34:34.278585 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.278523 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-h27h2_74defd5b-37e2-48c7-9b3c-849720978d12/kube-rbac-proxy-self/0.log" Jul 09 19:34:34.373215 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.373178 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-54jsv_72fec418-967a-4a13-84d1-2a0a141251be/node-exporter/0.log" Jul 09 19:34:34.401200 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.401181 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-54jsv_72fec418-967a-4a13-84d1-2a0a141251be/kube-rbac-proxy/0.log" Jul 09 19:34:34.421610 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.421589 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-54jsv_72fec418-967a-4a13-84d1-2a0a141251be/init-textfile/0.log" Jul 09 19:34:34.935739 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.935702 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b5f89968d-j7zhv_0abfb907-fdb2-40df-bc7f-fc5e3da9a86d/telemeter-client/0.log" Jul 09 19:34:34.962002 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.961977 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b5f89968d-j7zhv_0abfb907-fdb2-40df-bc7f-fc5e3da9a86d/reload/0.log" Jul 09 19:34:34.982971 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:34.982941 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b5f89968d-j7zhv_0abfb907-fdb2-40df-bc7f-fc5e3da9a86d/kube-rbac-proxy/0.log" Jul 09 19:34:35.416264 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.416236 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk"] Jul 09 19:34:35.416620 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.416602 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0094950f-5cd4-48a9-bb4f-d05967cf90cf" containerName="test-curl-7482-140307192413040" Jul 09 19:34:35.416620 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.416615 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0094950f-5cd4-48a9-bb4f-d05967cf90cf" containerName="test-curl-7482-140307192413040" Jul 09 19:34:35.416706 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.416694 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0094950f-5cd4-48a9-bb4f-d05967cf90cf" containerName="test-curl-7482-140307192413040" Jul 09 19:34:35.422019 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.422002 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.424387 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.424367 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fz6n6\"/\"default-dockercfg-pl4vb\"" Jul 09 19:34:35.424507 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.424408 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fz6n6\"/\"openshift-service-ca.crt\"" Jul 09 19:34:35.425298 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.425273 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fz6n6\"/\"kube-root-ca.crt\"" Jul 09 19:34:35.429621 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.428874 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk"] Jul 09 19:34:35.536541 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.536508 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpr2s\" (UniqueName: \"kubernetes.io/projected/9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f-kube-api-access-kpr2s\") pod \"perf-node-gather-daemonset-97pjk\" (UID: \"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.536541 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.536542 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f-lib-modules\") pod \"perf-node-gather-daemonset-97pjk\" (UID: \"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.536755 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.536567 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f-sys\") pod \"perf-node-gather-daemonset-97pjk\" (UID: \"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.536755 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.536626 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f-proc\") pod \"perf-node-gather-daemonset-97pjk\" (UID: \"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.536755 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.536714 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f-podres\") pod \"perf-node-gather-daemonset-97pjk\" (UID: \"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.637499 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.637463 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f-podres\") pod \"perf-node-gather-daemonset-97pjk\" (UID: \"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.637499 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.637504 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpr2s\" (UniqueName: \"kubernetes.io/projected/9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f-kube-api-access-kpr2s\") pod \"perf-node-gather-daemonset-97pjk\" (UID: \"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.637745 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.637525 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f-lib-modules\") pod \"perf-node-gather-daemonset-97pjk\" (UID: \"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.637745 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.637548 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f-sys\") pod \"perf-node-gather-daemonset-97pjk\" (UID: \"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.637745 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.637613 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f-sys\") pod \"perf-node-gather-daemonset-97pjk\" (UID: \"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.637745 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.637667 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f-lib-modules\") pod \"perf-node-gather-daemonset-97pjk\" (UID: \"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.637745 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.637684 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f-podres\") pod \"perf-node-gather-daemonset-97pjk\" (UID: \"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.637745 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.637713 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f-proc\") pod \"perf-node-gather-daemonset-97pjk\" (UID: \"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.638055 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.637811 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f-proc\") pod \"perf-node-gather-daemonset-97pjk\" (UID: \"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.645711 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.645694 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fz6n6\"/\"kube-root-ca.crt\"" Jul 09 19:34:35.656000 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.655975 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fz6n6\"/\"openshift-service-ca.crt\"" Jul 09 19:34:35.666086 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.666063 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpr2s\" (UniqueName: \"kubernetes.io/projected/9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f-kube-api-access-kpr2s\") pod \"perf-node-gather-daemonset-97pjk\" (UID: \"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.735979 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.735941 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fz6n6\"/\"default-dockercfg-pl4vb\"" Jul 09 19:34:35.744204 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.744188 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:35.879447 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:35.879425 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk"] Jul 09 19:34:35.881342 ip-10-0-133-185 kubenswrapper[2569]: W0709 19:34:35.881316 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9b213fd6_bd9e_4fa1_8fdb_1b77443ca92f.slice/crio-67a8588a75d709a9b7eba44665b0ec8aa69036c53f8d39e66135ba10027f31a7 WatchSource:0}: Error finding container 67a8588a75d709a9b7eba44665b0ec8aa69036c53f8d39e66135ba10027f31a7: Status 404 returned error can't find the container with id 67a8588a75d709a9b7eba44665b0ec8aa69036c53f8d39e66135ba10027f31a7 Jul 09 19:34:36.840253 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:36.840219 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" event={"ID":"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f","Type":"ContainerStarted","Data":"c2aa3f744ef51b647dcdc20dd44c66ac2769e472ab1e5192d2043f40a5831a63"} Jul 09 19:34:36.840667 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:36.840271 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" event={"ID":"9b213fd6-bd9e-4fa1-8fdb-1b77443ca92f","Type":"ContainerStarted","Data":"67a8588a75d709a9b7eba44665b0ec8aa69036c53f8d39e66135ba10027f31a7"} Jul 09 19:34:36.840667 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:36.840331 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:36.859046 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:36.859001 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" podStartSLOduration=1.858987562 podStartE2EDuration="1.858987562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:34:36.857041095 +0000 UTC m=+2461.682204132" watchObservedRunningTime="2026-07-09 19:34:36.858987562 +0000 UTC m=+2461.684150598" Jul 09 19:34:39.370893 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:39.370860 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lvppt_8f2246db-5e70-45a8-94a5-4f387631fe03/dns/0.log" Jul 09 19:34:39.394070 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:39.394046 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lvppt_8f2246db-5e70-45a8-94a5-4f387631fe03/kube-rbac-proxy/0.log" Jul 09 19:34:39.464914 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:39.464891 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v6rsk_dfcb4a64-02f6-451c-97c0-451804e2bd36/dns-node-resolver/0.log" Jul 09 19:34:40.025523 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:40.025489 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5dbb5989d9-7p7xp_88d1f57b-538f-4ef6-b0e8-616c9c493dab/registry/0.log" Jul 09 19:34:40.080754 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:40.080726 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bnbtq_ab56b93f-4a0a-4f46-9910-94df2f70621f/node-ca/0.log" Jul 09 19:34:41.315700 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:41.315672 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58cdb447-4kr8q_73e08385-5634-4158-928d-7c840cc3216b/kube-auth-proxy/0.log" Jul 09 19:34:41.442002 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:41.441971 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-nxrmk_c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52/payload-pre-processing/0.log" Jul 09 19:34:42.223164 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:42.223138 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qxft6_3bd52903-4c1f-4840-a92b-c1ff27ec76be/serve-healthcheck-canary/0.log" Jul 09 19:34:42.768510 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:42.768480 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4wknv_d9293675-16ef-4401-af79-aa74f9597e8f/kube-rbac-proxy/0.log" Jul 09 19:34:42.788589 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:42.788567 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4wknv_d9293675-16ef-4401-af79-aa74f9597e8f/exporter/0.log" Jul 09 19:34:42.810505 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:42.810483 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4wknv_d9293675-16ef-4401-af79-aa74f9597e8f/extractor/0.log" Jul 09 19:34:42.855051 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:42.855031 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-97pjk" Jul 09 19:34:45.697436 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:45.697386 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-698jk_20d3f3c6-078c-4b86-bd6d-d916d0f2cdf4/manager/0.log" Jul 09 19:34:46.124372 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:46.124343 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cd98cb899-p9bm8_11642f9a-aa84-47c8-b35d-1f4aa0397211/manager/0.log" Jul 09 19:34:46.146803 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:46.146776 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cd98cb899-p9bm8_11642f9a-aa84-47c8-b35d-1f4aa0397211/copy-manifests/0.log" Jul 09 19:34:46.308634 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:46.308590 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58cdb447-4kr8q_73e08385-5634-4158-928d-7c840cc3216b/kube-auth-proxy/0.log" Jul 09 19:34:46.409086 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:46.409036 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-nxrmk_c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52/payload-pre-processing/0.log" Jul 09 19:34:47.486479 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:47.486435 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-9dfc7f7f4-fnqqs_d21e2ff0-3336-4eb7-83b1-fc3a2212935f/manager/0.log" Jul 09 19:34:47.852875 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:47.852748 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58cdb447-4kr8q_73e08385-5634-4158-928d-7c840cc3216b/kube-auth-proxy/0.log" Jul 09 19:34:47.956743 ip-10-0-133-185 kubenswrapper[2569]: I0709 19:34:47.956714 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-nxrmk_c2ab4d25-34fa-4a94-9c0d-297e8b8f8b52/payload-pre-processing/0.log"