Jul 10 18:29:13.189639 ip-10-0-140-180 systemd[1]: Starting Kubernetes Kubelet... Jul 10 18:29:13.662385 ip-10-0-140-180 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 18:29:13.662385 ip-10-0-140-180 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 18:29:13.662385 ip-10-0-140-180 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 18:29:13.662385 ip-10-0-140-180 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 18:29:13.662385 ip-10-0-140-180 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 18:29:13.662385 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.645522 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 18:29:13.662385 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652629 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:29:13.662385 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652640 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:29:13.662385 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652644 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:29:13.662385 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652647 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:29:13.662385 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652650 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:29:13.662385 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652653 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:29:13.662385 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652656 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:29:13.662385 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652659 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652661 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652665 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652668 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652670 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652673 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652675 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652678 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652681 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652683 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652686 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652688 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652691 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652693 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652696 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652698 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652701 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652704 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652708 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652710 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:29:13.674254 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652713 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652716 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652719 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652722 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652725 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652728 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652731 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652733 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652736 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652740 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652744 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652748 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652750 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652753 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652756 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652760 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652763 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652767 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652770 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:29:13.676735 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652773 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652777 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652780 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652783 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652785 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652788 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652791 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652793 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652796 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652799 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652801 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652804 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652806 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652809 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652812 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652815 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652818 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652821 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652824 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652827 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:29:13.691174 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652830 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652832 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652835 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652837 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652840 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652843 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652847 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652851 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652854 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652857 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652860 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652863 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652866 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652868 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652871 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652874 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652876 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652879 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652881 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652884 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:29:13.693426 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652886 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652890 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652892 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652895 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652898 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652900 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652903 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652905 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652908 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652911 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652914 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652916 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.652919 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653637 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653644 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653646 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653649 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653652 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653655 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653657 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:29:13.695147 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653660 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653663 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653665 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653668 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653671 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653674 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653677 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653680 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653682 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653685 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653688 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653690 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653693 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653696 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653698 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653701 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653704 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653706 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653709 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653711 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:29:13.697871 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653714 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653717 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653719 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653722 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653724 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653727 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653729 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653731 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653734 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653736 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653739 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653742 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653744 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653747 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653751 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653755 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653758 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653761 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653764 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:29:13.699743 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653766 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653769 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653772 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653775 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653778 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653781 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653784 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653787 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653789 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653792 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653794 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653797 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653800 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653803 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653806 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653811 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653815 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653818 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653821 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:29:13.701893 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653824 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653826 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653829 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653832 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653835 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653837 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653840 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653843 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653846 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653848 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653851 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653854 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653857 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653860 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653862 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653865 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653868 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653870 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653873 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653875 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:29:13.703661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653879 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653881 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653884 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653887 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653889 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653892 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653894 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653897 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653899 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653902 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653904 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653907 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653910 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.653912 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654281 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654286 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654289 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654292 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654295 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654298 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:29:13.705967 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654301 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654304 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654306 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654309 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654312 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654315 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654318 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654320 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654323 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654325 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654328 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654333 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654336 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654338 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654341 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654343 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654346 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654368 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654371 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654376 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:29:13.707692 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654380 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654385 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654387 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654390 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654393 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654395 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654398 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654400 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654403 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654406 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654408 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654411 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654414 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654417 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654419 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654422 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654425 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654428 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654431 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654434 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:29:13.710047 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654436 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654439 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654441 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654445 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654448 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654451 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654454 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654457 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654460 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654462 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654465 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654467 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654470 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654474 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654476 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654479 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654483 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654487 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654489 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:29:13.711837 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654493 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654497 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654500 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654503 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654505 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654508 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654511 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654514 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654517 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654519 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654522 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654526 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654528 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654530 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654533 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654535 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654539 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654542 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654544 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654547 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:29:13.714141 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654549 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654552 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654555 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654557 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654559 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654562 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654566 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654568 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654571 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654573 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654576 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654578 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654581 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.654583 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654651 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654658 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654664 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654669 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654673 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654677 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654681 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 18:29:13.716109 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654685 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654689 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654692 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654696 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654699 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654703 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654706 2576 flags.go:64] FLAG: --cgroup-root="" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654710 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654713 2576 flags.go:64] FLAG: --client-ca-file="" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654716 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654718 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654722 2576 flags.go:64] FLAG: --cluster-domain="" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654725 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654728 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654731 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654735 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654738 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654742 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654745 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654749 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654752 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654754 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654757 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654761 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 18:29:13.718431 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654764 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654768 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654771 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654773 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654776 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654779 2576 flags.go:64] FLAG: --enable-server="true" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654782 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654787 2576 flags.go:64] FLAG: --event-burst="100" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654790 2576 flags.go:64] FLAG: --event-qps="50" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654793 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654796 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654799 2576 flags.go:64] FLAG: --eviction-hard="" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654803 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654806 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654809 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654814 2576 flags.go:64] FLAG: --eviction-soft="" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654817 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654820 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654823 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654825 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654828 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654831 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654834 2576 flags.go:64] FLAG: --feature-gates="" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654838 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654841 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 18:29:13.720388 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654845 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 18:29:13.719660 ip-10-0-140-180 systemd[1]: Started Kubernetes Kubelet. Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654848 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654851 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654855 2576 flags.go:64] FLAG: --help="false" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654857 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-140-180.ec2.internal" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654861 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654864 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654867 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654870 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654873 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654876 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654879 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654881 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654884 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654887 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654890 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654893 2576 flags.go:64] FLAG: --kube-reserved="" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654896 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654899 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654902 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654906 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654909 2576 flags.go:64] FLAG: --lock-file="" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654913 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654916 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 18:29:13.721879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654919 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654924 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654927 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654930 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654932 2576 flags.go:64] FLAG: --logging-format="text" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654935 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654938 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654941 2576 flags.go:64] FLAG: --manifest-url="" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654945 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654950 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654953 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654957 2576 flags.go:64] FLAG: --max-pods="110" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654960 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654963 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654966 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654969 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654972 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654975 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654979 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654985 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654989 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654991 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654994 2576 flags.go:64] FLAG: --pod-cidr="" Jul 10 18:29:13.924834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.654997 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655003 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655006 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655009 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655012 2576 flags.go:64] FLAG: --port="10250" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655015 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655018 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04ddd78b6580336a5" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655023 2576 flags.go:64] FLAG: --qos-reserved="" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655026 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655029 2576 flags.go:64] FLAG: --register-node="true" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655031 2576 flags.go:64] FLAG: --register-with-taints="" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655035 2576 flags.go:64] FLAG: --registry-burst="10" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655038 2576 flags.go:64] FLAG: --registry-qps="5" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655040 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655043 2576 flags.go:64] FLAG: --reserved-memory="" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655047 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655050 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655053 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655058 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655061 2576 flags.go:64] FLAG: --runonce="false" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655064 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655067 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655070 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655072 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655075 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655079 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 18:29:13.956644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655082 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655085 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655088 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655091 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655094 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655097 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655100 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655102 2576 flags.go:64] FLAG: --system-cgroups="" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655105 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655110 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655113 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655116 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655120 2576 flags.go:64] FLAG: --tls-min-version="" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655124 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655127 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655130 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655133 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655136 2576 flags.go:64] FLAG: --v="2" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655140 2576 flags.go:64] FLAG: --version="false" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655144 2576 flags.go:64] FLAG: --vmodule="" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655147 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.655150 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655262 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655266 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:29:13.976724 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655272 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655276 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655279 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655282 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655285 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655288 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655291 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655294 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655297 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655300 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655302 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655305 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655308 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655310 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655313 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655316 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655318 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655321 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655324 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655327 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:29:13.977665 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655329 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655333 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655336 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655338 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655341 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655343 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655346 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655361 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655364 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655367 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655370 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655373 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655376 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655379 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655382 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655384 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655387 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655390 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655392 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655395 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:29:13.978271 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655398 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655400 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655403 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655405 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655408 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655410 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655413 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655415 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655418 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655421 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655424 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655426 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655429 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655433 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655436 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655439 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655441 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655444 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655447 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:29:13.979068 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655451 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655455 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655457 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655460 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655462 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655466 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655469 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655472 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655474 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655478 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655482 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655485 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655488 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655491 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655494 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655497 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655500 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655502 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655505 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655508 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:29:13.979852 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655510 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655513 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655516 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655519 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655521 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655524 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655528 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655530 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655533 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655536 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655538 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655541 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655544 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655546 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655549 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655551 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655554 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:29:13.980654 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.655558 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:29:13.981195 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.656317 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 18:29:13.981195 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.664088 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 18:29:13.981195 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.664101 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 18:29:13.981195 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.664120 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 18:29:13.981195 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.664125 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 18:29:13.981195 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664180 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:29:13.981195 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664185 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:29:13.981195 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664188 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:29:13.981195 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664191 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:29:13.981195 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664196 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:29:13.981195 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664200 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:29:13.981195 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664203 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:29:13.981195 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664206 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:29:13.981195 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664209 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664212 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664216 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664218 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664221 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664224 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664227 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664230 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664232 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664235 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664238 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664241 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664244 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664247 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664249 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664252 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664254 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664257 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664259 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664262 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:29:13.981737 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664264 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664267 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664269 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664272 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664275 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664277 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664279 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664282 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664285 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664287 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664290 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664292 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664294 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664297 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664300 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664302 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664323 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664327 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664330 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664333 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:29:13.982333 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664337 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664339 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664342 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664386 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664401 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664405 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664410 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664412 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664415 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664419 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664421 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664424 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664427 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664430 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664433 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664435 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664438 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664440 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664443 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664446 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:29:13.982993 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664449 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664451 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664454 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664457 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664460 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664462 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664466 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664468 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664471 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664473 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664478 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664483 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664488 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664492 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664495 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664498 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664502 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664505 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664508 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:29:13.983606 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664511 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:29:13.984153 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664513 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:29:13.984153 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664516 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:29:13.984153 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664519 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:29:13.984153 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664522 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:29:13.984153 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664525 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:29:13.984153 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664528 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:29:13.984153 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664531 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:29:13.984153 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664534 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:29:13.984153 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664537 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:29:13.984153 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664539 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:29:13.984153 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664542 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:29:13.984153 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.664548 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 18:29:13.984153 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664672 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:29:13.984153 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664677 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:29:13.984153 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664680 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664683 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664686 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664689 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664691 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664694 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664697 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664700 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664703 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664705 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664708 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664711 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664714 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664716 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664719 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664722 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664725 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664729 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664732 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664734 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:29:13.984609 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664737 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664740 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664742 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664745 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664747 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664752 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664755 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664758 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664761 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664764 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664767 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664769 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664772 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664774 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664777 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664780 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664782 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664785 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664788 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:29:13.985184 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664793 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664796 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664799 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664801 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664804 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664810 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664813 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664816 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664818 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664822 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664824 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664827 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664829 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664832 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664834 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664837 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664839 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664842 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664845 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664847 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:29:13.985716 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664850 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664853 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664855 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664858 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664860 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664863 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664865 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664868 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664870 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664872 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664875 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664877 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664881 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664884 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664886 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664890 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664894 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664898 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664901 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:29:13.986281 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664904 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664907 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664910 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664913 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664915 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664918 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664920 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664923 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664925 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664928 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664930 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664933 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664936 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664938 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664941 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664943 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664946 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664949 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:29:13.986830 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:13.664951 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:29:13.987343 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.664956 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 18:29:13.987343 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.665052 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 18:29:13.987343 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.667484 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 18:29:13.987343 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.669013 2576 server.go:1019] "Starting client certificate rotation" Jul 10 18:29:13.987343 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.669108 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 18:29:13.987343 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.669142 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 18:29:13.987343 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.671774 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 18:29:13.987343 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.672885 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 18:29:13.987343 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.682127 2576 log.go:25] "Validated CRI v1 runtime API" Jul 10 18:29:13.987343 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.685140 2576 log.go:25] "Validated CRI v1 image API" Jul 10 18:29:13.987343 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.685972 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 18:29:13.987343 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.690575 2576 fs.go:135] Filesystem UUIDs: map[2d05f4fe-3ad5-469a-8edc-7ee57f8db36d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b1dea9ac-9be4-4032-b982-2b979aedf9a1:/dev/nvme0n1p3] Jul 10 18:29:13.987718 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.690591 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 18:29:13.987826 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.695795 2576 manager.go:216] Machine: {Timestamp:2026-07-10 18:29:13.695303716 +0000 UTC m=+0.373016169 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101035 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea8ae6f3fad0916d902f9f0222012 SystemUUID:ec2ea8ae-6f3f-ad09-16d9-02f9f0222012 BootID:7b25cf42-b3ac-4df7-a450-3c1c04fdc371 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c4:16:68:68:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c4:16:68:68:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:1e:48:bd:3b:34 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 18:29:13.987826 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.695894 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 18:29:13.987826 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.695965 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 18:29:13.987826 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.698670 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 18:29:13.987826 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.698699 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-180.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 18:29:13.987826 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.699579 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 18:29:13.987826 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.699592 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 18:29:13.987826 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.699600 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.699614 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.702792 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.706750 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.706886 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.707630 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.707645 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.707658 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.707686 2576 kubelet.go:398] "Adding apiserver pod source" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.707696 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.707989 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.708003 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.710189 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.712601 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.712633 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.712802 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.712817 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.712826 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.712834 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.712845 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.712852 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.713459 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.713468 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.713478 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.713487 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.713498 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.713508 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.715329 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.715343 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.718564 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.718829 2576 server.go:1268] "Started kubelet" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.719622 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.719666 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.719630 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.723732 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.725442 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.725615 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-180.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.726033 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.729304 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-180.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 18:29:13.988453 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.729655 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.729266 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-180.ec2.internal.18c1011d856084fd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-180.ec2.internal,UID:ip-10-0-140-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-180.ec2.internal,},FirstTimestamp:2026-07-10 18:29:13.718801661 +0000 UTC m=+0.396514116,LastTimestamp:2026-07-10 18:29:13.718801661 +0000 UTC m=+0.396514116,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-180.ec2.internal,}" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.731967 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.731986 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.732044 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.732704 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.732717 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.732704 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.732846 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.732856 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.732858 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.732873 2576 factory.go:55] Registering systemd factory Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.732881 2576 factory.go:223] Registration of the systemd container factory successfully Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.733160 2576 factory.go:153] Registering CRI-O factory Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.733172 2576 factory.go:223] Registration of the crio container factory successfully Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.733194 2576 factory.go:103] Registering Raw factory Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.733217 2576 manager.go:1196] Started watching for new ooms in manager Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.733636 2576 manager.go:349] Starting recovery of all containers Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.736104 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.736172 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-180.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.744301 2576 manager.go:354] Recovery completed Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.748805 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.750609 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.750651 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.750662 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.751729 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.751742 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.751753 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.751760 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.752519 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.752555 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.752568 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.752857 2576 policy_none.go:49] "None policy: Start" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.752940 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 18:29:14.148312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.752954 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 18:29:14.151867 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.753185 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-180.ec2.internal.18c1011d87462de1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-180.ec2.internal,UID:ip-10-0-140-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-180.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-180.ec2.internal,},FirstTimestamp:2026-07-10 18:29:13.750629857 +0000 UTC m=+0.428342311,LastTimestamp:2026-07-10 18:29:13.750629857 +0000 UTC m=+0.428342311,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-180.ec2.internal,}" Jul 10 18:29:14.151867 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.753910 2576 policy_none.go:47] "Start" Jul 10 18:29:14.151867 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.763093 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-180.ec2.internal.18c1011d874695bd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-180.ec2.internal,UID:ip-10-0-140-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-180.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-180.ec2.internal,},FirstTimestamp:2026-07-10 18:29:13.750656445 +0000 UTC m=+0.428368897,LastTimestamp:2026-07-10 18:29:13.750656445 +0000 UTC m=+0.428368897,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-180.ec2.internal,}" Jul 10 18:29:14.151867 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.768642 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-180.ec2.internal.18c1011d8746ba5a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-180.ec2.internal,UID:ip-10-0-140-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-180.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-180.ec2.internal,},FirstTimestamp:2026-07-10 18:29:13.750665818 +0000 UTC m=+0.428378273,LastTimestamp:2026-07-10 18:29:13.750665818 +0000 UTC m=+0.428378273,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-180.ec2.internal,}" Jul 10 18:29:14.151867 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.773371 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4hpzg" Jul 10 18:29:14.151867 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.774162 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-180.ec2.internal.18c1011d87462de1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-180.ec2.internal.18c1011d87462de1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-180.ec2.internal,UID:ip-10-0-140-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-180.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-180.ec2.internal,},FirstTimestamp:2026-07-10 18:29:13.750629857 +0000 UTC m=+0.428342311,LastTimestamp:2026-07-10 18:29:13.752539892 +0000 UTC m=+0.430252348,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-180.ec2.internal,}" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.778575 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-180.ec2.internal.18c1011d874695bd\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-180.ec2.internal.18c1011d874695bd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-180.ec2.internal,UID:ip-10-0-140-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-180.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-180.ec2.internal,},FirstTimestamp:2026-07-10 18:29:13.750656445 +0000 UTC m=+0.428368897,LastTimestamp:2026-07-10 18:29:13.752561934 +0000 UTC m=+0.430274389,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-180.ec2.internal,}" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.781301 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4hpzg" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.782844 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-180.ec2.internal.18c1011d8746ba5a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-180.ec2.internal.18c1011d8746ba5a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-180.ec2.internal,UID:ip-10-0-140-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-180.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-180.ec2.internal,},FirstTimestamp:2026-07-10 18:29:13.750665818 +0000 UTC m=+0.428378273,LastTimestamp:2026-07-10 18:29:13.752573379 +0000 UTC m=+0.430285834,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-180.ec2.internal,}" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.805021 2576 manager.go:337] "Starting Device Plugin manager" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.805053 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.805066 2576 server.go:87] "Starting device plugin registration server" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.805314 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.805324 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.805406 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.805504 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.805524 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.805950 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.805993 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.835768 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.837001 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.837042 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.837063 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.837104 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.839629 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.905671 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.906640 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.906676 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.906689 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.906712 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.152149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.919671 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.919688 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-180.ec2.internal\": node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.937696 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-180.ec2.internal"] Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.937749 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.939635 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.939654 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.939664 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.940678 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.940866 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.940896 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.941419 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.941434 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.941443 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.941756 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.941774 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.941782 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.942364 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.942383 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.943011 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.943027 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:13.943036 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.980439 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:13.997307 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-180.ec2.internal\" not found" node="ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:14.001759 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-180.ec2.internal\" not found" node="ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.035396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/34ce888784a2bd4fe2722ae68893f984-config\") pod \"kube-apiserver-proxy-ip-10-0-140-180.ec2.internal\" (UID: \"34ce888784a2bd4fe2722ae68893f984\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.035416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4df75909386e31b15f53ca980eafa97d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal\" (UID: \"4df75909386e31b15f53ca980eafa97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.152850 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.035433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4df75909386e31b15f53ca980eafa97d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal\" (UID: \"4df75909386e31b15f53ca980eafa97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.153635 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:14.081409 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:14.153635 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.136294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/34ce888784a2bd4fe2722ae68893f984-config\") pod \"kube-apiserver-proxy-ip-10-0-140-180.ec2.internal\" (UID: \"34ce888784a2bd4fe2722ae68893f984\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.153635 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.136321 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4df75909386e31b15f53ca980eafa97d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal\" (UID: \"4df75909386e31b15f53ca980eafa97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.153635 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.136338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4df75909386e31b15f53ca980eafa97d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal\" (UID: \"4df75909386e31b15f53ca980eafa97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.153635 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.136405 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4df75909386e31b15f53ca980eafa97d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal\" (UID: \"4df75909386e31b15f53ca980eafa97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.153635 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.136422 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4df75909386e31b15f53ca980eafa97d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal\" (UID: \"4df75909386e31b15f53ca980eafa97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.153635 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.136405 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/34ce888784a2bd4fe2722ae68893f984-config\") pod \"kube-apiserver-proxy-ip-10-0-140-180.ec2.internal\" (UID: \"34ce888784a2bd4fe2722ae68893f984\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.306223 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:14.182297 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:14.306223 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:14.282664 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:14.306223 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.298834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.306223 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.304751 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-180.ec2.internal" Jul 10 18:29:14.519454 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:14.383677 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:14.519454 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:14.484095 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:14.725236 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:14.585166 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:14.725236 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.651577 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:29:14.725236 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.668341 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 18:29:14.725236 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.668467 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 18:29:14.725236 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.668510 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 18:29:14.725236 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:14.685907 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:14.933892 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.746118 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:29:14.933892 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.775002 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:29:14.933892 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.776057 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:29:14.933892 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.776084 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:29:14.933892 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.776093 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:29:14.933892 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.785112 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 18:24:14 +0000 UTC" deadline="2027-12-23 07:10:20.005016971 +0000 UTC" Jul 10 18:29:14.933892 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.785133 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12732h41m5.219886508s" Jul 10 18:29:14.933892 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:14.786275 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:14.933892 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:14.886554 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:15.127006 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:14.958366 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df75909386e31b15f53ca980eafa97d.slice/crio-ccb15fa0c6fdeb6617825b0f62f9b0c5a1c0a2a10c55ea1a1b28010a3cb01837 WatchSource:0}: Error finding container ccb15fa0c6fdeb6617825b0f62f9b0c5a1c0a2a10c55ea1a1b28010a3cb01837: Status 404 returned error can't find the container with id ccb15fa0c6fdeb6617825b0f62f9b0c5a1c0a2a10c55ea1a1b28010a3cb01837 Jul 10 18:29:15.127006 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:14.958825 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ce888784a2bd4fe2722ae68893f984.slice/crio-cb91305aabfaef00da759278185aa570a4305dcc022699cb1fa33c7facf5ae3d WatchSource:0}: Error finding container cb91305aabfaef00da759278185aa570a4305dcc022699cb1fa33c7facf5ae3d: Status 404 returned error can't find the container with id cb91305aabfaef00da759278185aa570a4305dcc022699cb1fa33c7facf5ae3d Jul 10 18:29:15.127006 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:14.964591 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 18:29:15.127006 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:14.987483 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:15.127006 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:15.088674 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:15.308629 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:15.189644 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-180.ec2.internal\" not found" Jul 10 18:29:15.308629 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.196800 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:29:15.308629 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.233138 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal" Jul 10 18:29:15.308629 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.254790 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 18:29:15.308629 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.256057 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-180.ec2.internal" Jul 10 18:29:15.308629 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.266954 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.709970 2576 apiserver.go:52] "Watching apiserver" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.718782 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.720587 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-z28ln","kube-system/konnectivity-agent-g54w6","kube-system/kube-apiserver-proxy-ip-10-0-140-180.ec2.internal","openshift-dns/node-resolver-gnzkb","openshift-multus/multus-additional-cni-plugins-94n7z","openshift-multus/network-metrics-daemon-xlffx","openshift-network-diagnostics/network-check-target-lqrqz","openshift-ovn-kubernetes/ovnkube-node-fzct8","kube-system/global-pull-secret-syncer-rs2hc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x","openshift-cluster-node-tuning-operator/tuned-ncspq","openshift-image-registry/node-ca-b5dpv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal","openshift-multus/multus-jd9kf"] Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.722570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:15.722665 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.724246 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.725925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.726770 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vptlv\"" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.727188 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.727761 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gnzkb" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.728175 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.728419 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.728488 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.729174 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dzwvx\"" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.729567 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.729822 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.729936 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.730437 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.730591 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vmbrh\"" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.731903 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:15.731969 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.732364 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.732458 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.732867 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7kb7b\"" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.733074 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 18:29:15.794751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.734506 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:15.734568 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.742154 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.742518 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.743865 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g54w6" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.745887 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.746180 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.746196 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.746227 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.746240 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x88xd\"" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.746768 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.746819 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.746770 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.746964 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.747438 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hdd7r\"" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.748950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-run-multus-certs\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.748975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgcjf\" (UniqueName: \"kubernetes.io/projected/91835427-78a0-464d-87ab-d70b45a77388-kube-api-access-rgcjf\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.748993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwg8w\" (UniqueName: \"kubernetes.io/projected/43310758-62c8-4c7b-8c6b-4fe08aa348d1-kube-api-access-qwg8w\") pod \"network-metrics-daemon-xlffx\" (UID: \"43310758-62c8-4c7b-8c6b-4fe08aa348d1\") " pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749008 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-sys\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ee9d9500-f51f-4369-a156-76087731ddc0-etc-tuned\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749069 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e18304c4-c506-4566-a4d1-997d4720acf6-cni-binary-copy\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-var-lib-cni-bin\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.797589 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749163 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z28ln" Jul 10 18:29:15.798790 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749179 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91835427-78a0-464d-87ab-d70b45a77388-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.798790 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:15.798790 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749240 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftwz9\" (UniqueName: \"kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9\") pod \"network-check-target-lqrqz\" (UID: \"74645fb8-cd43-4907-b0dc-3446ed74822b\") " pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:15.798790 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-var-lib-kubelet\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.798790 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749296 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-var-lib-cni-multus\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.798790 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749321 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec189118-e07f-4ba5-a5c0-cc1910849ca6-kubelet-config\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:15.798790 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749347 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee9d9500-f51f-4369-a156-76087731ddc0-tmp\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.798790 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-multus-socket-dir-parent\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.798790 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749435 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-kubernetes\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.798790 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-multus-cni-dir\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.798790 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-run-k8s-cni-cncf-io\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.798790 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-etc-kubernetes\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.798790 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24f9d007-8b0e-4e0f-8062-3a66675027e8-hosts-file\") pod \"node-resolver-gnzkb\" (UID: \"24f9d007-8b0e-4e0f-8062-3a66675027e8\") " pod="openshift-dns/node-resolver-gnzkb" Jul 10 18:29:15.798790 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec189118-e07f-4ba5-a5c0-cc1910849ca6-dbus\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749632 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-modprobe-d\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-os-release\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749705 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-run-netns\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749747 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbx8c\" (UniqueName: \"kubernetes.io/projected/e18304c4-c506-4566-a4d1-997d4720acf6-kube-api-access-gbx8c\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749774 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749838 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749772 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91835427-78a0-464d-87ab-d70b45a77388-cni-binary-copy\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs\") pod \"network-metrics-daemon-xlffx\" (UID: \"43310758-62c8-4c7b-8c6b-4fe08aa348d1\") " pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749926 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b5rxp\"" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749930 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-sysctl-conf\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-run\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.749994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-var-lib-kubelet\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7c8l\" (UniqueName: \"kubernetes.io/projected/ee9d9500-f51f-4369-a156-76087731ddc0-kube-api-access-c7c8l\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750025 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750069 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-system-cni-dir\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.800187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-hostroot\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.801234 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hm8f\" (UniqueName: \"kubernetes.io/projected/24f9d007-8b0e-4e0f-8062-3a66675027e8-kube-api-access-8hm8f\") pod \"node-resolver-gnzkb\" (UID: \"24f9d007-8b0e-4e0f-8062-3a66675027e8\") " pod="openshift-dns/node-resolver-gnzkb" Jul 10 18:29:15.801234 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91835427-78a0-464d-87ab-d70b45a77388-system-cni-dir\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.801234 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-sysconfig\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.801234 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750211 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-lib-modules\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.801234 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-multus-conf-dir\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.801234 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f9d007-8b0e-4e0f-8062-3a66675027e8-tmp-dir\") pod \"node-resolver-gnzkb\" (UID: \"24f9d007-8b0e-4e0f-8062-3a66675027e8\") " pod="openshift-dns/node-resolver-gnzkb" Jul 10 18:29:15.801234 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750316 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91835427-78a0-464d-87ab-d70b45a77388-cnibin\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.801234 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/91835427-78a0-464d-87ab-d70b45a77388-os-release\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.801234 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91835427-78a0-464d-87ab-d70b45a77388-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.801234 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750422 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91835427-78a0-464d-87ab-d70b45a77388-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.801234 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750445 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-host\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.801234 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-sysctl-d\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.801234 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-systemd\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.801234 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-cnibin\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.805176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750565 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 18:29:15.805176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750632 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e18304c4-c506-4566-a4d1-997d4720acf6-multus-daemon-config\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.805176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750756 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b5dpv" Jul 10 18:29:15.805176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.750933 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 18:29:15.805176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.752971 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 18:29:15.805176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.753180 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:29:15.805176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.753245 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 18:29:15.805176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.753320 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 18:29:15.805176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.753534 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cmcdb\"" Jul 10 18:29:15.805176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.753573 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tg28n\"" Jul 10 18:29:15.805176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.754035 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 18:29:15.805176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.754971 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 18:29:15.805176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.787182 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rj5l2" Jul 10 18:29:15.805176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.804701 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rj5l2" Jul 10 18:29:15.878021 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.833990 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 18:29:15.878021 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.843450 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-180.ec2.internal" event={"ID":"34ce888784a2bd4fe2722ae68893f984","Type":"ContainerStarted","Data":"cb91305aabfaef00da759278185aa570a4305dcc022699cb1fa33c7facf5ae3d"} Jul 10 18:29:15.878021 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.846349 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal" event={"ID":"4df75909386e31b15f53ca980eafa97d","Type":"ContainerStarted","Data":"ccb15fa0c6fdeb6617825b0f62f9b0c5a1c0a2a10c55ea1a1b28010a3cb01837"} Jul 10 18:29:15.878021 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.851849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-cnibin\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.878021 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.851890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9e0fea4f-bfdb-45e6-8244-c61e85e5d347-iptables-alerter-script\") pod \"iptables-alerter-z28ln\" (UID: \"9e0fea4f-bfdb-45e6-8244-c61e85e5d347\") " pod="openshift-network-operator/iptables-alerter-z28ln" Jul 10 18:29:15.878021 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.851975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-systemd-units\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.878021 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.851994 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-cnibin\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.878021 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ee9d9500-f51f-4369-a156-76087731ddc0-etc-tuned\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.878021 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-etc-selinux\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:15.878021 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-run-openvswitch\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.878021 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-run-ovn\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.878021 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852106 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-log-socket\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.878021 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.878021 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-var-lib-kubelet\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.878021 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-var-lib-cni-multus\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.881122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec189118-e07f-4ba5-a5c0-cc1910849ca6-kubelet-config\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:15.881122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852237 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-cni-netd\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.881122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-multus-socket-dir-parent\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.881122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-sys-fs\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:15.881122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852331 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ldl\" (UniqueName: \"kubernetes.io/projected/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-kube-api-access-b7ldl\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:15.881122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7b0a074-7ac6-4e87-9667-f1d0e4920459-serviceca\") pod \"node-ca-b5dpv\" (UID: \"b7b0a074-7ac6-4e87-9667-f1d0e4920459\") " pod="openshift-image-registry/node-ca-b5dpv" Jul 10 18:29:15.881122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-run-netns\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.881122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852376 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 18:29:15.881122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852548 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-kubernetes\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.881122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-kubernetes\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.881122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-multus-cni-dir\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.881122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec189118-e07f-4ba5-a5c0-cc1910849ca6-kubelet-config\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:15.881122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-etc-kubernetes\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.881122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852706 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-var-lib-cni-multus\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.881122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec189118-e07f-4ba5-a5c0-cc1910849ca6-dbus\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:15.882087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-var-lib-kubelet\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.882087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-ovn-node-metrics-cert\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.882087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-modprobe-d\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.882087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852810 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-multus-socket-dir-parent\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.882087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-os-release\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.882087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbx8c\" (UniqueName: \"kubernetes.io/projected/e18304c4-c506-4566-a4d1-997d4720acf6-kube-api-access-gbx8c\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.882087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852878 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-etc-kubernetes\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.882087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91835427-78a0-464d-87ab-d70b45a77388-cni-binary-copy\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.882087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852897 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-multus-cni-dir\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.882087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852901 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec189118-e07f-4ba5-a5c0-cc1910849ca6-dbus\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:15.882087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.852979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-os-release\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.882087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853001 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-modprobe-d\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.882087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853010 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-socket-dir\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:15.882087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-ovnkube-config\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.882087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/91835427-78a0-464d-87ab-d70b45a77388-os-release\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.993349 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91835427-78a0-464d-87ab-d70b45a77388-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.993349 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91835427-78a0-464d-87ab-d70b45a77388-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.993349 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-sysctl-conf\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.993349 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-hostroot\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.993349 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853297 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/91835427-78a0-464d-87ab-d70b45a77388-os-release\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.993349 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853321 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hm8f\" (UniqueName: \"kubernetes.io/projected/24f9d007-8b0e-4e0f-8062-3a66675027e8-kube-api-access-8hm8f\") pod \"node-resolver-gnzkb\" (UID: \"24f9d007-8b0e-4e0f-8062-3a66675027e8\") " pod="openshift-dns/node-resolver-gnzkb" Jul 10 18:29:15.993349 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853351 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf2qw\" (UniqueName: \"kubernetes.io/projected/9e0fea4f-bfdb-45e6-8244-c61e85e5d347-kube-api-access-wf2qw\") pod \"iptables-alerter-z28ln\" (UID: \"9e0fea4f-bfdb-45e6-8244-c61e85e5d347\") " pod="openshift-network-operator/iptables-alerter-z28ln" Jul 10 18:29:15.993349 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853379 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.993349 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f9d007-8b0e-4e0f-8062-3a66675027e8-tmp-dir\") pod \"node-resolver-gnzkb\" (UID: \"24f9d007-8b0e-4e0f-8062-3a66675027e8\") " pod="openshift-dns/node-resolver-gnzkb" Jul 10 18:29:15.993349 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91835427-78a0-464d-87ab-d70b45a77388-cnibin\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.993349 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwg8w\" (UniqueName: \"kubernetes.io/projected/43310758-62c8-4c7b-8c6b-4fe08aa348d1-kube-api-access-qwg8w\") pod \"network-metrics-daemon-xlffx\" (UID: \"43310758-62c8-4c7b-8c6b-4fe08aa348d1\") " pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:15.993349 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e0fea4f-bfdb-45e6-8244-c61e85e5d347-host-slash\") pod \"iptables-alerter-z28ln\" (UID: \"9e0fea4f-bfdb-45e6-8244-c61e85e5d347\") " pod="openshift-network-operator/iptables-alerter-z28ln" Jul 10 18:29:15.993349 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-host\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.993349 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-sysctl-d\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.994703 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e18304c4-c506-4566-a4d1-997d4720acf6-multus-daemon-config\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.994703 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-run-multus-certs\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.994703 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853657 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgcjf\" (UniqueName: \"kubernetes.io/projected/91835427-78a0-464d-87ab-d70b45a77388-kube-api-access-rgcjf\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.994703 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853680 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91835427-78a0-464d-87ab-d70b45a77388-cni-binary-copy\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.994703 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-registration-dir\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:15.994703 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853766 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-host\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.994703 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853820 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91835427-78a0-464d-87ab-d70b45a77388-cnibin\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.994703 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853822 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-sysctl-d\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.994703 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853836 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91835427-78a0-464d-87ab-d70b45a77388-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.994703 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853904 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-run-multus-certs\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.994703 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.853971 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f9d007-8b0e-4e0f-8062-3a66675027e8-tmp-dir\") pod \"node-resolver-gnzkb\" (UID: \"24f9d007-8b0e-4e0f-8062-3a66675027e8\") " pod="openshift-dns/node-resolver-gnzkb" Jul 10 18:29:15.994703 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-var-lib-openvswitch\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.994703 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854131 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-sys\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.994703 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e18304c4-c506-4566-a4d1-997d4720acf6-cni-binary-copy\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.995335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854191 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-var-lib-cni-bin\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.995335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwz9\" (UniqueName: \"kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9\") pod \"network-check-target-lqrqz\" (UID: \"74645fb8-cd43-4907-b0dc-3446ed74822b\") " pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:15.995335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854246 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcb2s\" (UniqueName: \"kubernetes.io/projected/b7b0a074-7ac6-4e87-9667-f1d0e4920459-kube-api-access-hcb2s\") pod \"node-ca-b5dpv\" (UID: \"b7b0a074-7ac6-4e87-9667-f1d0e4920459\") " pod="openshift-image-registry/node-ca-b5dpv" Jul 10 18:29:15.995335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854273 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-etc-openvswitch\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.995335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-cni-bin\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.995335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854332 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-device-dir\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:15.995335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854356 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-slash\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.995335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854376 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e18304c4-c506-4566-a4d1-997d4720acf6-multus-daemon-config\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.995335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854417 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee9d9500-f51f-4369-a156-76087731ddc0-tmp\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.995335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854448 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-sysctl-conf\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.995335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r578g\" (UniqueName: \"kubernetes.io/projected/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-kube-api-access-r578g\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.995335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854492 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91835427-78a0-464d-87ab-d70b45a77388-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.995335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854499 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-hostroot\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.995335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-run-k8s-cni-cncf-io\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.996225 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24f9d007-8b0e-4e0f-8062-3a66675027e8-hosts-file\") pod \"node-resolver-gnzkb\" (UID: \"24f9d007-8b0e-4e0f-8062-3a66675027e8\") " pod="openshift-dns/node-resolver-gnzkb" Jul 10 18:29:15.996225 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-kubelet\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.996225 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854537 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-run-k8s-cni-cncf-io\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.996225 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854593 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-node-log\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.996225 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-run-netns\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.996225 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854636 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24f9d007-8b0e-4e0f-8062-3a66675027e8-hosts-file\") pod \"node-resolver-gnzkb\" (UID: \"24f9d007-8b0e-4e0f-8062-3a66675027e8\") " pod="openshift-dns/node-resolver-gnzkb" Jul 10 18:29:15.996225 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs\") pod \"network-metrics-daemon-xlffx\" (UID: \"43310758-62c8-4c7b-8c6b-4fe08aa348d1\") " pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:15.996225 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854669 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-env-overrides\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.996225 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19914451-ba36-4c62-a151-f74fc24b9102-konnectivity-ca\") pod \"konnectivity-agent-g54w6\" (UID: \"19914451-ba36-4c62-a151-f74fc24b9102\") " pod="kube-system/konnectivity-agent-g54w6" Jul 10 18:29:15.996225 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854695 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-run-netns\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.996225 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:15.996225 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:15.996225 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-run\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.996225 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-var-lib-kubelet\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.996225 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:15.854801 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:29:15.997062 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7c8l\" (UniqueName: \"kubernetes.io/projected/ee9d9500-f51f-4369-a156-76087731ddc0-kube-api-access-c7c8l\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.997062 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-system-cni-dir\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.997062 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91835427-78a0-464d-87ab-d70b45a77388-system-cni-dir\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.997062 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854892 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7b0a074-7ac6-4e87-9667-f1d0e4920459-host\") pod \"node-ca-b5dpv\" (UID: \"b7b0a074-7ac6-4e87-9667-f1d0e4920459\") " pod="openshift-image-registry/node-ca-b5dpv" Jul 10 18:29:15.997062 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854894 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e18304c4-c506-4566-a4d1-997d4720acf6-cni-binary-copy\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.997062 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-sysconfig\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.997062 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854472 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-sys\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.997062 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.854982 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-host-var-lib-cni-bin\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.997062 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:15.855029 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs podName:43310758-62c8-4c7b-8c6b-4fe08aa348d1 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:16.355005724 +0000 UTC m=+3.032718182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs") pod "network-metrics-daemon-xlffx" (UID: "43310758-62c8-4c7b-8c6b-4fe08aa348d1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:29:15.997062 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855033 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91835427-78a0-464d-87ab-d70b45a77388-system-cni-dir\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.997062 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:15.855098 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:29:15.997062 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-lib-modules\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.997062 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:15.855153 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret podName:ec189118-e07f-4ba5-a5c0-cc1910849ca6 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:16.355141004 +0000 UTC m=+3.032853449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret") pod "global-pull-secret-syncer-rs2hc" (UID: "ec189118-e07f-4ba5-a5c0-cc1910849ca6") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:29:15.997062 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-multus-conf-dir\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.997796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855099 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-sysconfig\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.997796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855245 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-var-lib-kubelet\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.997796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855245 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-lib-modules\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.997796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855276 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91835427-78a0-464d-87ab-d70b45a77388-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.997796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855299 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-system-cni-dir\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.997796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-run-systemd\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.997796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855342 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-run\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.997796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-ovnkube-script-lib\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.997796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855420 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19914451-ba36-4c62-a151-f74fc24b9102-agent-certs\") pod \"konnectivity-agent-g54w6\" (UID: \"19914451-ba36-4c62-a151-f74fc24b9102\") " pod="kube-system/konnectivity-agent-g54w6" Jul 10 18:29:15.997796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-systemd\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.997796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855505 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91835427-78a0-464d-87ab-d70b45a77388-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.997796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e18304c4-c506-4566-a4d1-997d4720acf6-multus-conf-dir\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.997796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.855565 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ee9d9500-f51f-4369-a156-76087731ddc0-etc-systemd\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.997796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.858585 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ee9d9500-f51f-4369-a156-76087731ddc0-etc-tuned\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.997796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.859586 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee9d9500-f51f-4369-a156-76087731ddc0-tmp\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.998521 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:15.867186 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:29:15.998521 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:15.867207 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:29:15.998521 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:15.867221 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ftwz9 for pod openshift-network-diagnostics/network-check-target-lqrqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:29:15.998521 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:15.867284 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9 podName:74645fb8-cd43-4907-b0dc-3446ed74822b nodeName:}" failed. No retries permitted until 2026-07-10 18:29:16.367265932 +0000 UTC m=+3.044978377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ftwz9" (UniqueName: "kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9") pod "network-check-target-lqrqz" (UID: "74645fb8-cd43-4907-b0dc-3446ed74822b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:29:15.998521 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.869559 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbx8c\" (UniqueName: \"kubernetes.io/projected/e18304c4-c506-4566-a4d1-997d4720acf6-kube-api-access-gbx8c\") pod \"multus-jd9kf\" (UID: \"e18304c4-c506-4566-a4d1-997d4720acf6\") " pod="openshift-multus/multus-jd9kf" Jul 10 18:29:15.998521 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.873830 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgcjf\" (UniqueName: \"kubernetes.io/projected/91835427-78a0-464d-87ab-d70b45a77388-kube-api-access-rgcjf\") pod \"multus-additional-cni-plugins-94n7z\" (UID: \"91835427-78a0-464d-87ab-d70b45a77388\") " pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:15.998521 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.875983 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7c8l\" (UniqueName: \"kubernetes.io/projected/ee9d9500-f51f-4369-a156-76087731ddc0-kube-api-access-c7c8l\") pod \"tuned-ncspq\" (UID: \"ee9d9500-f51f-4369-a156-76087731ddc0\") " pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:15.998521 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.876402 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwg8w\" (UniqueName: \"kubernetes.io/projected/43310758-62c8-4c7b-8c6b-4fe08aa348d1-kube-api-access-qwg8w\") pod \"network-metrics-daemon-xlffx\" (UID: \"43310758-62c8-4c7b-8c6b-4fe08aa348d1\") " pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:15.998521 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.878349 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hm8f\" (UniqueName: \"kubernetes.io/projected/24f9d007-8b0e-4e0f-8062-3a66675027e8-kube-api-access-8hm8f\") pod \"node-resolver-gnzkb\" (UID: \"24f9d007-8b0e-4e0f-8062-3a66675027e8\") " pod="openshift-dns/node-resolver-gnzkb" Jul 10 18:29:15.998521 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.955950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r578g\" (UniqueName: \"kubernetes.io/projected/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-kube-api-access-r578g\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.998521 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.955989 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-kubelet\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.998521 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-node-log\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.998521 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-env-overrides\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.998521 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19914451-ba36-4c62-a151-f74fc24b9102-konnectivity-ca\") pod \"konnectivity-agent-g54w6\" (UID: \"19914451-ba36-4c62-a151-f74fc24b9102\") " pod="kube-system/konnectivity-agent-g54w6" Jul 10 18:29:15.999373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956113 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:15.999373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956180 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-node-log\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.999373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956203 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:15.999373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956518 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-kubelet\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.999373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7b0a074-7ac6-4e87-9667-f1d0e4920459-host\") pod \"node-ca-b5dpv\" (UID: \"b7b0a074-7ac6-4e87-9667-f1d0e4920459\") " pod="openshift-image-registry/node-ca-b5dpv" Jul 10 18:29:15.999373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-run-systemd\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.999373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-ovnkube-script-lib\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.999373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19914451-ba36-4c62-a151-f74fc24b9102-agent-certs\") pod \"konnectivity-agent-g54w6\" (UID: \"19914451-ba36-4c62-a151-f74fc24b9102\") " pod="kube-system/konnectivity-agent-g54w6" Jul 10 18:29:15.999373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9e0fea4f-bfdb-45e6-8244-c61e85e5d347-iptables-alerter-script\") pod \"iptables-alerter-z28ln\" (UID: \"9e0fea4f-bfdb-45e6-8244-c61e85e5d347\") " pod="openshift-network-operator/iptables-alerter-z28ln" Jul 10 18:29:15.999373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-systemd-units\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.999373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956722 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-etc-selinux\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:15.999373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-run-openvswitch\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.999373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-run-ovn\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:15.999373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956794 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19914451-ba36-4c62-a151-f74fc24b9102-konnectivity-ca\") pod \"konnectivity-agent-g54w6\" (UID: \"19914451-ba36-4c62-a151-f74fc24b9102\") " pod="kube-system/konnectivity-agent-g54w6" Jul 10 18:29:15.999373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-log-socket\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.000475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.000475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-cni-netd\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.000475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956887 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-sys-fs\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:16.000475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ldl\" (UniqueName: \"kubernetes.io/projected/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-kube-api-access-b7ldl\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:16.000475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7b0a074-7ac6-4e87-9667-f1d0e4920459-serviceca\") pod \"node-ca-b5dpv\" (UID: \"b7b0a074-7ac6-4e87-9667-f1d0e4920459\") " pod="openshift-image-registry/node-ca-b5dpv" Jul 10 18:29:16.000475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-run-netns\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.000475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.956992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-ovn-node-metrics-cert\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.000475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-socket-dir\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:16.000475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-ovnkube-config\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.000475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf2qw\" (UniqueName: \"kubernetes.io/projected/9e0fea4f-bfdb-45e6-8244-c61e85e5d347-kube-api-access-wf2qw\") pod \"iptables-alerter-z28ln\" (UID: \"9e0fea4f-bfdb-45e6-8244-c61e85e5d347\") " pod="openshift-network-operator/iptables-alerter-z28ln" Jul 10 18:29:16.000475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.000475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e0fea4f-bfdb-45e6-8244-c61e85e5d347-host-slash\") pod \"iptables-alerter-z28ln\" (UID: \"9e0fea4f-bfdb-45e6-8244-c61e85e5d347\") " pod="openshift-network-operator/iptables-alerter-z28ln" Jul 10 18:29:16.000475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-registration-dir\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:16.000475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-var-lib-openvswitch\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957204 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-env-overrides\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcb2s\" (UniqueName: \"kubernetes.io/projected/b7b0a074-7ac6-4e87-9667-f1d0e4920459-kube-api-access-hcb2s\") pod \"node-ca-b5dpv\" (UID: \"b7b0a074-7ac6-4e87-9667-f1d0e4920459\") " pod="openshift-image-registry/node-ca-b5dpv" Jul 10 18:29:16.001210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957259 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-var-lib-openvswitch\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-etc-openvswitch\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957297 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7b0a074-7ac6-4e87-9667-f1d0e4920459-host\") pod \"node-ca-b5dpv\" (UID: \"b7b0a074-7ac6-4e87-9667-f1d0e4920459\") " pod="openshift-image-registry/node-ca-b5dpv" Jul 10 18:29:16.001210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-cni-bin\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957337 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-run-systemd\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-device-dir\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:16.001210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.957724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-slash\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.961060 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-systemd-units\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.961173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9e0fea4f-bfdb-45e6-8244-c61e85e5d347-iptables-alerter-script\") pod \"iptables-alerter-z28ln\" (UID: \"9e0fea4f-bfdb-45e6-8244-c61e85e5d347\") " pod="openshift-network-operator/iptables-alerter-z28ln" Jul 10 18:29:16.001210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.961237 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-etc-selinux\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:16.001210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.961372 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-run-openvswitch\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.961465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-run-ovn\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.961524 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-log-socket\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.961594 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.961656 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-cni-netd\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.961683 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-ovnkube-config\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.961735 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-sys-fs\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:16.001992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.961799 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.962086 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7b0a074-7ac6-4e87-9667-f1d0e4920459-serviceca\") pod \"node-ca-b5dpv\" (UID: \"b7b0a074-7ac6-4e87-9667-f1d0e4920459\") " pod="openshift-image-registry/node-ca-b5dpv" Jul 10 18:29:16.001992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.962197 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-socket-dir\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:16.001992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.962256 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e0fea4f-bfdb-45e6-8244-c61e85e5d347-host-slash\") pod \"iptables-alerter-z28ln\" (UID: \"9e0fea4f-bfdb-45e6-8244-c61e85e5d347\") " pod="openshift-network-operator/iptables-alerter-z28ln" Jul 10 18:29:16.001992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.962316 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-run-netns\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.962385 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-registration-dir\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:16.001992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.962450 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-cni-bin\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.963024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-device-dir\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:16.001992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.963035 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-etc-openvswitch\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.001992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.963228 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-host-slash\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.002630 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.963636 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-ovnkube-script-lib\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.002630 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.965410 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-ovn-node-metrics-cert\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.002630 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.966811 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19914451-ba36-4c62-a151-f74fc24b9102-agent-certs\") pod \"konnectivity-agent-g54w6\" (UID: \"19914451-ba36-4c62-a151-f74fc24b9102\") " pod="kube-system/konnectivity-agent-g54w6" Jul 10 18:29:16.002630 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.968631 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r578g\" (UniqueName: \"kubernetes.io/projected/bf1d25c8-e207-43d4-8a1d-6db4146f16eb-kube-api-access-r578g\") pod \"ovnkube-node-fzct8\" (UID: \"bf1d25c8-e207-43d4-8a1d-6db4146f16eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.002630 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.971882 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf2qw\" (UniqueName: \"kubernetes.io/projected/9e0fea4f-bfdb-45e6-8244-c61e85e5d347-kube-api-access-wf2qw\") pod \"iptables-alerter-z28ln\" (UID: \"9e0fea4f-bfdb-45e6-8244-c61e85e5d347\") " pod="openshift-network-operator/iptables-alerter-z28ln" Jul 10 18:29:16.002630 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.975120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ldl\" (UniqueName: \"kubernetes.io/projected/ce7b5caf-80c1-42a4-8ab1-5eab8a137fab-kube-api-access-b7ldl\") pod \"aws-ebs-csi-driver-node-8fg7x\" (UID: \"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:16.002630 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:15.975420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcb2s\" (UniqueName: \"kubernetes.io/projected/b7b0a074-7ac6-4e87-9667-f1d0e4920459-kube-api-access-hcb2s\") pod \"node-ca-b5dpv\" (UID: \"b7b0a074-7ac6-4e87-9667-f1d0e4920459\") " pod="openshift-image-registry/node-ca-b5dpv" Jul 10 18:29:16.037675 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.037641 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ncspq" Jul 10 18:29:16.050488 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.050463 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jd9kf" Jul 10 18:29:16.058888 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:16.058851 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9d9500_f51f_4369_a156_76087731ddc0.slice/crio-4fe798df50aa555088b25978d890661db6bbbd5650152cb9f09e94e9ffac337d WatchSource:0}: Error finding container 4fe798df50aa555088b25978d890661db6bbbd5650152cb9f09e94e9ffac337d: Status 404 returned error can't find the container with id 4fe798df50aa555088b25978d890661db6bbbd5650152cb9f09e94e9ffac337d Jul 10 18:29:16.060151 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.060053 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gnzkb" Jul 10 18:29:16.066844 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.066824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-94n7z" Jul 10 18:29:16.074612 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.074590 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:16.082272 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.082206 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g54w6" Jul 10 18:29:16.082272 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:16.082237 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f9d007_8b0e_4e0f_8062_3a66675027e8.slice/crio-384cdfa9ccce9b9e69b06ada924a5f92a562b044faed715540383c620c44243d WatchSource:0}: Error finding container 384cdfa9ccce9b9e69b06ada924a5f92a562b044faed715540383c620c44243d: Status 404 returned error can't find the container with id 384cdfa9ccce9b9e69b06ada924a5f92a562b044faed715540383c620c44243d Jul 10 18:29:16.088607 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.088586 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" Jul 10 18:29:16.092086 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:16.092059 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91835427_78a0_464d_87ab_d70b45a77388.slice/crio-14df538629edc95a1c7d400505479c0ed1f35f826661c04f37e67a1b69070df1 WatchSource:0}: Error finding container 14df538629edc95a1c7d400505479c0ed1f35f826661c04f37e67a1b69070df1: Status 404 returned error can't find the container with id 14df538629edc95a1c7d400505479c0ed1f35f826661c04f37e67a1b69070df1 Jul 10 18:29:16.098397 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.098154 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z28ln" Jul 10 18:29:16.105031 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.104846 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b5dpv" Jul 10 18:29:16.105240 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:16.105216 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19914451_ba36_4c62_a151_f74fc24b9102.slice/crio-ddf58a3da0843f14e544ff6e4b3760c4f6a2a2c8d88cc2ef3d3e2f8f5fe32047 WatchSource:0}: Error finding container ddf58a3da0843f14e544ff6e4b3760c4f6a2a2c8d88cc2ef3d3e2f8f5fe32047: Status 404 returned error can't find the container with id ddf58a3da0843f14e544ff6e4b3760c4f6a2a2c8d88cc2ef3d3e2f8f5fe32047 Jul 10 18:29:16.163849 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.163817 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:29:16.361300 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.361213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs\") pod \"network-metrics-daemon-xlffx\" (UID: \"43310758-62c8-4c7b-8c6b-4fe08aa348d1\") " pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:16.361300 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.361247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:16.361525 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:16.361392 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:29:16.361525 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:16.361446 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:29:16.361525 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:16.361461 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs podName:43310758-62c8-4c7b-8c6b-4fe08aa348d1 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:17.361443021 +0000 UTC m=+4.039155486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs") pod "network-metrics-daemon-xlffx" (UID: "43310758-62c8-4c7b-8c6b-4fe08aa348d1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:29:16.361525 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:16.361496 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret podName:ec189118-e07f-4ba5-a5c0-cc1910849ca6 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:17.361479396 +0000 UTC m=+4.039191838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret") pod "global-pull-secret-syncer-rs2hc" (UID: "ec189118-e07f-4ba5-a5c0-cc1910849ca6") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:29:16.428434 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.428402 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:29:16.462170 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.462130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwz9\" (UniqueName: \"kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9\") pod \"network-check-target-lqrqz\" (UID: \"74645fb8-cd43-4907-b0dc-3446ed74822b\") " pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:16.462391 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:16.462344 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:29:16.462391 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:16.462396 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:29:16.462588 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:16.462415 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ftwz9 for pod openshift-network-diagnostics/network-check-target-lqrqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:29:16.462588 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:16.462490 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9 podName:74645fb8-cd43-4907-b0dc-3446ed74822b nodeName:}" failed. No retries permitted until 2026-07-10 18:29:17.462467359 +0000 UTC m=+4.140179837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ftwz9" (UniqueName: "kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9") pod "network-check-target-lqrqz" (UID: "74645fb8-cd43-4907-b0dc-3446ed74822b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:29:16.482083 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:16.482056 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1d25c8_e207_43d4_8a1d_6db4146f16eb.slice/crio-37a0df0868363d6150467eda7d98a6a7bb074fe6007b3b2d9f6c0f9cc2093526 WatchSource:0}: Error finding container 37a0df0868363d6150467eda7d98a6a7bb074fe6007b3b2d9f6c0f9cc2093526: Status 404 returned error can't find the container with id 37a0df0868363d6150467eda7d98a6a7bb074fe6007b3b2d9f6c0f9cc2093526 Jul 10 18:29:16.808065 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.807895 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:24:16 +0000 UTC" deadline="2028-01-14 07:52:55.317432701 +0000 UTC" Jul 10 18:29:16.808065 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.807942 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13261h23m38.509496967s" Jul 10 18:29:16.837704 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.837670 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:16.837878 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.837723 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:16.837878 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:16.837812 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:16.838046 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:16.837991 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:16.849844 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.849805 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94n7z" event={"ID":"91835427-78a0-464d-87ab-d70b45a77388","Type":"ContainerStarted","Data":"14df538629edc95a1c7d400505479c0ed1f35f826661c04f37e67a1b69070df1"} Jul 10 18:29:16.851451 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.851423 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gnzkb" event={"ID":"24f9d007-8b0e-4e0f-8062-3a66675027e8","Type":"ContainerStarted","Data":"384cdfa9ccce9b9e69b06ada924a5f92a562b044faed715540383c620c44243d"} Jul 10 18:29:16.852992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.852967 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jd9kf" event={"ID":"e18304c4-c506-4566-a4d1-997d4720acf6","Type":"ContainerStarted","Data":"e7ab6a2f26742714c45bc4018bb9e57bb2cc1a3c821f4fa7a0f0ef0ddecbd7b2"} Jul 10 18:29:16.854483 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.854452 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ncspq" event={"ID":"ee9d9500-f51f-4369-a156-76087731ddc0","Type":"ContainerStarted","Data":"4fe798df50aa555088b25978d890661db6bbbd5650152cb9f09e94e9ffac337d"} Jul 10 18:29:16.856404 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.856360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" event={"ID":"bf1d25c8-e207-43d4-8a1d-6db4146f16eb","Type":"ContainerStarted","Data":"37a0df0868363d6150467eda7d98a6a7bb074fe6007b3b2d9f6c0f9cc2093526"} Jul 10 18:29:16.857724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:16.857677 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g54w6" event={"ID":"19914451-ba36-4c62-a151-f74fc24b9102","Type":"ContainerStarted","Data":"ddf58a3da0843f14e544ff6e4b3760c4f6a2a2c8d88cc2ef3d3e2f8f5fe32047"} Jul 10 18:29:17.371379 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:17.371335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs\") pod \"network-metrics-daemon-xlffx\" (UID: \"43310758-62c8-4c7b-8c6b-4fe08aa348d1\") " pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:17.371594 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:17.371411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:17.371594 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:17.371584 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:29:17.371786 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:17.371650 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret podName:ec189118-e07f-4ba5-a5c0-cc1910849ca6 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:19.371630183 +0000 UTC m=+6.049342631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret") pod "global-pull-secret-syncer-rs2hc" (UID: "ec189118-e07f-4ba5-a5c0-cc1910849ca6") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:29:17.372091 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:17.372069 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:29:17.372187 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:17.372128 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs podName:43310758-62c8-4c7b-8c6b-4fe08aa348d1 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:19.372110657 +0000 UTC m=+6.049823100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs") pod "network-metrics-daemon-xlffx" (UID: "43310758-62c8-4c7b-8c6b-4fe08aa348d1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:29:17.473407 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:17.472702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwz9\" (UniqueName: \"kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9\") pod \"network-check-target-lqrqz\" (UID: \"74645fb8-cd43-4907-b0dc-3446ed74822b\") " pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:17.473407 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:17.472941 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:29:17.473407 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:17.472959 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:29:17.473407 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:17.472973 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ftwz9 for pod openshift-network-diagnostics/network-check-target-lqrqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:29:17.473407 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:17.473033 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9 podName:74645fb8-cd43-4907-b0dc-3446ed74822b nodeName:}" failed. No retries permitted until 2026-07-10 18:29:19.473013051 +0000 UTC m=+6.150725500 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ftwz9" (UniqueName: "kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9") pod "network-check-target-lqrqz" (UID: "74645fb8-cd43-4907-b0dc-3446ed74822b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:29:17.809843 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:17.806797 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7b5caf_80c1_42a4_8ab1_5eab8a137fab.slice/crio-531bcd7d9ebaeb05c0034a29cb1e149f3450b46c2095627f91ec8a5b8021055d WatchSource:0}: Error finding container 531bcd7d9ebaeb05c0034a29cb1e149f3450b46c2095627f91ec8a5b8021055d: Status 404 returned error can't find the container with id 531bcd7d9ebaeb05c0034a29cb1e149f3450b46c2095627f91ec8a5b8021055d Jul 10 18:29:17.809843 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:17.809033 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:24:16 +0000 UTC" deadline="2027-12-19 20:15:54.607497444 +0000 UTC" Jul 10 18:29:17.809843 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:17.809067 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12649h46m36.798435857s" Jul 10 18:29:17.839848 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:17.839822 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:17.839984 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:17.839952 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:17.864711 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:17.864678 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b5dpv" event={"ID":"b7b0a074-7ac6-4e87-9667-f1d0e4920459","Type":"ContainerStarted","Data":"27a1b284882e1659161f795c0754694e7a0aa29cb184ff8d883863fc9ca8e134"} Jul 10 18:29:17.867889 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:17.867856 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z28ln" event={"ID":"9e0fea4f-bfdb-45e6-8244-c61e85e5d347","Type":"ContainerStarted","Data":"49d57fe386e8a8cfd2b0595e0efef2fac67217afb3ae3cd06bcabab0c142b05b"} Jul 10 18:29:17.880892 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:17.880782 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" event={"ID":"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab","Type":"ContainerStarted","Data":"531bcd7d9ebaeb05c0034a29cb1e149f3450b46c2095627f91ec8a5b8021055d"} Jul 10 18:29:18.838381 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:18.838111 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:18.838998 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:18.838517 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:18.838998 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:18.838971 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:18.839126 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:18.839104 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:18.890243 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:18.890201 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-180.ec2.internal" event={"ID":"34ce888784a2bd4fe2722ae68893f984","Type":"ContainerStarted","Data":"1581b302e524e98bd571869460e9ee4054382ec7affa1c6bfd421504aea6acad"} Jul 10 18:29:18.897840 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:18.897808 2576 generic.go:356] "Generic (PLEG): container finished" podID="4df75909386e31b15f53ca980eafa97d" containerID="e54eae1c122fbd8e93bf5bb6f85d4840fd1272e2e57a1e8aef5c5501faec2eb5" exitCode=0 Jul 10 18:29:18.897972 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:18.897857 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal" event={"ID":"4df75909386e31b15f53ca980eafa97d","Type":"ContainerDied","Data":"e54eae1c122fbd8e93bf5bb6f85d4840fd1272e2e57a1e8aef5c5501faec2eb5"} Jul 10 18:29:18.934508 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:18.934429 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-180.ec2.internal" podStartSLOduration=3.934406848 podStartE2EDuration="3.934406848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:18.910531324 +0000 UTC m=+5.588243791" watchObservedRunningTime="2026-07-10 18:29:18.934406848 +0000 UTC m=+5.612119305" Jul 10 18:29:19.388100 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:19.388060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs\") pod \"network-metrics-daemon-xlffx\" (UID: \"43310758-62c8-4c7b-8c6b-4fe08aa348d1\") " pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:19.388283 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:19.388118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:19.388362 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:19.388309 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:29:19.388555 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:19.388377 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret podName:ec189118-e07f-4ba5-a5c0-cc1910849ca6 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:23.388357905 +0000 UTC m=+10.066070350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret") pod "global-pull-secret-syncer-rs2hc" (UID: "ec189118-e07f-4ba5-a5c0-cc1910849ca6") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:29:19.388555 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:19.388516 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:29:19.388687 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:19.388572 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs podName:43310758-62c8-4c7b-8c6b-4fe08aa348d1 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:23.388556186 +0000 UTC m=+10.066268646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs") pod "network-metrics-daemon-xlffx" (UID: "43310758-62c8-4c7b-8c6b-4fe08aa348d1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:29:19.489337 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:19.489296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwz9\" (UniqueName: \"kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9\") pod \"network-check-target-lqrqz\" (UID: \"74645fb8-cd43-4907-b0dc-3446ed74822b\") " pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:19.489591 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:19.489558 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:29:19.489591 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:19.489588 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:29:19.489718 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:19.489612 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ftwz9 for pod openshift-network-diagnostics/network-check-target-lqrqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:29:19.489718 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:19.489674 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9 podName:74645fb8-cd43-4907-b0dc-3446ed74822b nodeName:}" failed. No retries permitted until 2026-07-10 18:29:23.489652818 +0000 UTC m=+10.167365282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ftwz9" (UniqueName: "kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9") pod "network-check-target-lqrqz" (UID: "74645fb8-cd43-4907-b0dc-3446ed74822b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:29:19.837507 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:19.837371 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:19.837669 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:19.837511 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:20.837972 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:20.837928 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:20.838756 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:20.837928 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:20.838756 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:20.838090 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:20.838756 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:20.838133 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:21.837564 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:21.837526 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:21.837776 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:21.837692 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:22.838232 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:22.838185 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:22.838232 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:22.838217 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:22.838741 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:22.838325 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:22.838816 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:22.838751 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:23.424325 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:23.424110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs\") pod \"network-metrics-daemon-xlffx\" (UID: \"43310758-62c8-4c7b-8c6b-4fe08aa348d1\") " pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:23.424325 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:23.424154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:23.424325 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:23.424266 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:29:23.424325 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:23.424280 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:29:23.424655 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:23.424340 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret podName:ec189118-e07f-4ba5-a5c0-cc1910849ca6 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:31.424323033 +0000 UTC m=+18.102035487 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret") pod "global-pull-secret-syncer-rs2hc" (UID: "ec189118-e07f-4ba5-a5c0-cc1910849ca6") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:29:23.424655 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:23.424374 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs podName:43310758-62c8-4c7b-8c6b-4fe08aa348d1 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:31.424364366 +0000 UTC m=+18.102076806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs") pod "network-metrics-daemon-xlffx" (UID: "43310758-62c8-4c7b-8c6b-4fe08aa348d1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:29:23.526343 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:23.525536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwz9\" (UniqueName: \"kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9\") pod \"network-check-target-lqrqz\" (UID: \"74645fb8-cd43-4907-b0dc-3446ed74822b\") " pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:23.526343 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:23.525739 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:29:23.526343 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:23.525756 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:29:23.526343 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:23.525769 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ftwz9 for pod openshift-network-diagnostics/network-check-target-lqrqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:29:23.526343 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:23.525825 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9 podName:74645fb8-cd43-4907-b0dc-3446ed74822b nodeName:}" failed. No retries permitted until 2026-07-10 18:29:31.525806605 +0000 UTC m=+18.203519050 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ftwz9" (UniqueName: "kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9") pod "network-check-target-lqrqz" (UID: "74645fb8-cd43-4907-b0dc-3446ed74822b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:29:23.838881 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:23.838611 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:23.838881 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:23.838724 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:24.837372 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:24.837283 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:24.837517 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:24.837430 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:24.837517 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:24.837473 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:24.837639 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:24.837581 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:25.838331 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:25.838297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:25.838808 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:25.838471 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:26.837966 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:26.837935 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:26.837966 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:26.837975 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:26.838155 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:26.838054 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:26.838221 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:26.838171 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:27.837750 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:27.837712 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:27.838221 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:27.837842 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:28.837307 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:28.837271 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:28.837307 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:28.837299 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:28.837538 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:28.837431 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:28.837611 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:28.837566 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:29.837914 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:29.837886 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:29.838289 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:29.837993 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:30.837886 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:30.837857 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:30.838135 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:30.837857 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:30.838135 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:30.837985 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:30.838135 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:30.838065 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:31.483176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:31.483138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs\") pod \"network-metrics-daemon-xlffx\" (UID: \"43310758-62c8-4c7b-8c6b-4fe08aa348d1\") " pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:31.483176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:31.483175 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:31.483424 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:31.483306 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:29:31.483424 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:31.483395 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs podName:43310758-62c8-4c7b-8c6b-4fe08aa348d1 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:47.48337268 +0000 UTC m=+34.161085136 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs") pod "network-metrics-daemon-xlffx" (UID: "43310758-62c8-4c7b-8c6b-4fe08aa348d1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:29:31.483424 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:31.483311 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:29:31.483559 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:31.483455 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret podName:ec189118-e07f-4ba5-a5c0-cc1910849ca6 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:47.483438298 +0000 UTC m=+34.161150756 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret") pod "global-pull-secret-syncer-rs2hc" (UID: "ec189118-e07f-4ba5-a5c0-cc1910849ca6") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:29:31.584230 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:31.584173 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwz9\" (UniqueName: \"kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9\") pod \"network-check-target-lqrqz\" (UID: \"74645fb8-cd43-4907-b0dc-3446ed74822b\") " pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:31.584433 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:31.584329 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:29:31.584433 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:31.584366 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:29:31.584433 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:31.584380 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ftwz9 for pod openshift-network-diagnostics/network-check-target-lqrqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:29:31.584605 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:31.584438 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9 podName:74645fb8-cd43-4907-b0dc-3446ed74822b nodeName:}" failed. No retries permitted until 2026-07-10 18:29:47.584420067 +0000 UTC m=+34.262132518 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ftwz9" (UniqueName: "kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9") pod "network-check-target-lqrqz" (UID: "74645fb8-cd43-4907-b0dc-3446ed74822b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:29:31.838189 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:31.838154 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:31.838652 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:31.838293 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:32.838005 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:32.837971 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:32.838186 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:32.837983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:32.838186 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:32.838090 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:32.838186 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:32.838169 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:33.838075 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:33.838042 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:33.838241 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:33.838140 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:34.837824 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:34.837796 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:34.838146 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:34.837797 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:34.838146 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:34.837881 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:34.838146 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:34.837965 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:35.837915 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.837580 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:35.838707 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:35.837997 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:35.933183 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.933151 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jd9kf" event={"ID":"e18304c4-c506-4566-a4d1-997d4720acf6","Type":"ContainerStarted","Data":"7656d95dc9229c4f248f1d05ed2c630f70cfd65b9ebcaecff0ed43ac51e6720b"} Jul 10 18:29:35.934646 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.934622 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ncspq" event={"ID":"ee9d9500-f51f-4369-a156-76087731ddc0","Type":"ContainerStarted","Data":"56f5739778dcea225fd5d8e1030b82f6a336a06d32ebef09f581026d30f2029c"} Jul 10 18:29:35.935900 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.935880 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b5dpv" event={"ID":"b7b0a074-7ac6-4e87-9667-f1d0e4920459","Type":"ContainerStarted","Data":"40767401f39ad1d847dbb85734a52201a9d6029305152693e3c9cb08f8117998"} Jul 10 18:29:35.937149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.937130 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" event={"ID":"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab","Type":"ContainerStarted","Data":"ba57f46c4ff2e3ec45639abf8655d0b5cc921b9ec674023fc10c07379443aea5"} Jul 10 18:29:35.938802 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.938784 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal" event={"ID":"4df75909386e31b15f53ca980eafa97d","Type":"ContainerStarted","Data":"375b390028a2d70203af2cd2f238dd56a66575b2fb6e8bdc7a075f229464215c"} Jul 10 18:29:35.943174 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.943157 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:29:35.943514 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.943498 2576 generic.go:356] "Generic (PLEG): container finished" podID="bf1d25c8-e207-43d4-8a1d-6db4146f16eb" containerID="0678b273ee2f10ffb99d8444eff61043c1852b192371231b7c8f0fc55c65be52" exitCode=1 Jul 10 18:29:35.943591 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.943558 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" event={"ID":"bf1d25c8-e207-43d4-8a1d-6db4146f16eb","Type":"ContainerStarted","Data":"f34dc823eb51fe60b140ef0c46af6a26f17b79b9cad3f0405fe2196daed74e9a"} Jul 10 18:29:35.943648 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.943630 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" event={"ID":"bf1d25c8-e207-43d4-8a1d-6db4146f16eb","Type":"ContainerStarted","Data":"b2990c89eebdf338a0faf8768cdab8716278c44b4872577f4b032a8eb2d05691"} Jul 10 18:29:35.943698 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.943684 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" event={"ID":"bf1d25c8-e207-43d4-8a1d-6db4146f16eb","Type":"ContainerStarted","Data":"15dbc285146622574335b964991473eaa171d1581caffc9a097c80ad301e2c3a"} Jul 10 18:29:35.943752 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.943736 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" event={"ID":"bf1d25c8-e207-43d4-8a1d-6db4146f16eb","Type":"ContainerStarted","Data":"11a289a0c4ed5cd58075f56f8d58a6b1189568577fd78559febfd655d1c88421"} Jul 10 18:29:35.943805 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.943774 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" event={"ID":"bf1d25c8-e207-43d4-8a1d-6db4146f16eb","Type":"ContainerDied","Data":"0678b273ee2f10ffb99d8444eff61043c1852b192371231b7c8f0fc55c65be52"} Jul 10 18:29:35.943856 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.943824 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" event={"ID":"bf1d25c8-e207-43d4-8a1d-6db4146f16eb","Type":"ContainerStarted","Data":"1e8d709d5e6a99bd1163998eb05e1d6ca8f77f01835ad76bb3ee62ae04e7cd38"} Jul 10 18:29:35.944822 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.944803 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g54w6" event={"ID":"19914451-ba36-4c62-a151-f74fc24b9102","Type":"ContainerStarted","Data":"1c056a4e9754fd65baaf6c2ea0101594c9374d54d2dcb1526ecb128e53c12b17"} Jul 10 18:29:35.946132 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.946113 2576 generic.go:356] "Generic (PLEG): container finished" podID="91835427-78a0-464d-87ab-d70b45a77388" containerID="5585013c04f488188a83397727a1c0410d4abd1e48894be860e8043c8344857b" exitCode=0 Jul 10 18:29:35.946227 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.946185 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94n7z" event={"ID":"91835427-78a0-464d-87ab-d70b45a77388","Type":"ContainerDied","Data":"5585013c04f488188a83397727a1c0410d4abd1e48894be860e8043c8344857b"} Jul 10 18:29:35.947476 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.947454 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gnzkb" event={"ID":"24f9d007-8b0e-4e0f-8062-3a66675027e8","Type":"ContainerStarted","Data":"88a49901719d3f26cbea4a761beb7627f6e4af907f29cb53cdeccade43999309"} Jul 10 18:29:35.954557 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.954523 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jd9kf" podStartSLOduration=2.907356218 podStartE2EDuration="21.954511708s" totalImagesPullingTime="19.04715549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:35.954450871 +0000 UTC m=+22.632163333" watchObservedRunningTime="2026-07-10 18:29:35.954511708 +0000 UTC m=+22.632224171" Jul 10 18:29:35.974015 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:35.973978 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ncspq" podStartSLOduration=2.961449264 podStartE2EDuration="21.973967504s" totalImagesPullingTime="19.01251824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:35.973472459 +0000 UTC m=+22.651184921" watchObservedRunningTime="2026-07-10 18:29:35.973967504 +0000 UTC m=+22.651679966" Jul 10 18:29:36.014765 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:36.014715 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gnzkb" podStartSLOduration=3.297690081 podStartE2EDuration="22.014697174s" totalImagesPullingTime="18.717007093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:35.99290474 +0000 UTC m=+22.670617196" watchObservedRunningTime="2026-07-10 18:29:36.014697174 +0000 UTC m=+22.692409636" Jul 10 18:29:36.054555 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:36.054500 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-b5dpv" podStartSLOduration=5.114881764 podStartE2EDuration="22.054482784s" totalImagesPullingTime="16.93960102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:36.054070979 +0000 UTC m=+22.731783442" watchObservedRunningTime="2026-07-10 18:29:36.054482784 +0000 UTC m=+22.732195251" Jul 10 18:29:36.054807 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:36.054781 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g54w6" podStartSLOduration=3.103682103 podStartE2EDuration="22.054773092s" totalImagesPullingTime="18.951090989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:36.014777009 +0000 UTC m=+22.692489471" watchObservedRunningTime="2026-07-10 18:29:36.054773092 +0000 UTC m=+22.732485555" Jul 10 18:29:36.082591 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:36.082511 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-180.ec2.internal" podStartSLOduration=21.08249624 podStartE2EDuration="21.08249624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:36.081728395 +0000 UTC m=+22.759440848" watchObservedRunningTime="2026-07-10 18:29:36.08249624 +0000 UTC m=+22.760208701" Jul 10 18:29:36.645688 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:36.645653 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 18:29:36.817790 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:36.817697 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T18:29:36.645698455Z","UUID":"f525486e-d2f8-40ec-aa92-2dbf14597e06","Handler":null,"Name":"","Endpoint":""} Jul 10 18:29:36.819336 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:36.819313 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 18:29:36.819472 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:36.819376 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 18:29:36.837947 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:36.837920 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:36.838380 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:36.837931 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:36.838380 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:36.838036 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:36.838380 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:36.838142 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:36.951281 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:36.951238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z28ln" event={"ID":"9e0fea4f-bfdb-45e6-8244-c61e85e5d347","Type":"ContainerStarted","Data":"60095c7631018998a672297f1bf02f65a039d42a2cb2667927693adca194cde2"} Jul 10 18:29:36.953454 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:36.953422 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" event={"ID":"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab","Type":"ContainerStarted","Data":"211e3b2c79b935451b9ca33959c1dca593f6c3993dece9f1c003e13a49bb1d42"} Jul 10 18:29:36.967181 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:36.967137 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z28ln" podStartSLOduration=5.75764086 podStartE2EDuration="22.967124111s" totalImagesPullingTime="17.209483251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:36.967059211 +0000 UTC m=+23.644771672" watchObservedRunningTime="2026-07-10 18:29:36.967124111 +0000 UTC m=+23.644836577" Jul 10 18:29:37.838124 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:37.838101 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:37.838506 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:37.838237 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:37.957789 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:37.957754 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" event={"ID":"ce7b5caf-80c1-42a4-8ab1-5eab8a137fab","Type":"ContainerStarted","Data":"e0d8453494db480df1f7b86ed12789469f9c1cd176b75697638e38a058756606"} Jul 10 18:29:37.984782 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:37.984739 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8fg7x" podStartSLOduration=4.427172772 podStartE2EDuration="23.984718786s" totalImagesPullingTime="19.557546014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:29:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:37.984684702 +0000 UTC m=+24.662397176" watchObservedRunningTime="2026-07-10 18:29:37.984718786 +0000 UTC m=+24.662431243" Jul 10 18:29:38.838137 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:38.838108 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:38.838502 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:38.838107 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:38.838502 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:38.838211 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:38.838502 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:38.838275 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:38.963123 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:38.963098 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:29:38.963396 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:38.963374 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" event={"ID":"bf1d25c8-e207-43d4-8a1d-6db4146f16eb","Type":"ContainerStarted","Data":"585a2b0136ed824d00b0ebf0ff14991dc2a03ff6491387dfc8ff49dd98ab3a3f"} Jul 10 18:29:39.044047 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:39.044019 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g54w6" Jul 10 18:29:39.044534 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:39.044515 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g54w6" Jul 10 18:29:39.838551 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:39.838299 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:39.838974 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:39.838659 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:39.965727 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:39.965697 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g54w6" Jul 10 18:29:39.966259 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:39.966236 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g54w6" Jul 10 18:29:40.838104 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:40.838074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:40.838282 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:40.838074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:40.838282 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:40.838185 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:40.838282 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:40.838275 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:41.838526 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:41.838277 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:41.839211 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:41.838635 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:41.970507 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:41.970475 2576 generic.go:356] "Generic (PLEG): container finished" podID="91835427-78a0-464d-87ab-d70b45a77388" containerID="3b8e2d182cb40aeadb67fd6c8a3d3a1e7777dc5a9a14254f43bc72ca73fbc797" exitCode=0 Jul 10 18:29:41.970691 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:41.970571 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94n7z" event={"ID":"91835427-78a0-464d-87ab-d70b45a77388","Type":"ContainerDied","Data":"3b8e2d182cb40aeadb67fd6c8a3d3a1e7777dc5a9a14254f43bc72ca73fbc797"} Jul 10 18:29:41.973757 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:41.973740 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:29:41.974135 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:41.974112 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" event={"ID":"bf1d25c8-e207-43d4-8a1d-6db4146f16eb","Type":"ContainerStarted","Data":"bda4bc259f73092506aea7f4834f2277fcef9c7374ec6d61ce656548074dc69d"} Jul 10 18:29:41.974546 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:41.974532 2576 scope.go:117] "RemoveContainer" containerID="0678b273ee2f10ffb99d8444eff61043c1852b192371231b7c8f0fc55c65be52" Jul 10 18:29:42.838180 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:42.838150 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:42.838397 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:42.838259 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:42.838397 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:42.838314 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:42.838516 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:42.838456 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:42.979251 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:42.979228 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:29:42.979631 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:42.979606 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" event={"ID":"bf1d25c8-e207-43d4-8a1d-6db4146f16eb","Type":"ContainerStarted","Data":"a7f7121040f57fea987adc9599be5098d8dc5401dc4e9b7948fa85280ac4c465"} Jul 10 18:29:42.979898 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:42.979879 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:42.979974 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:42.979951 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:42.980025 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:42.979999 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:42.997247 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:42.997227 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:42.997367 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:42.997330 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:29:43.000882 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:43.000861 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rs2hc"] Jul 10 18:29:43.000965 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:43.000949 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:43.001043 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:43.001027 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:43.003992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:43.003965 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lqrqz"] Jul 10 18:29:43.004069 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:43.004028 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:43.004108 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:43.004090 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:43.009407 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:43.009387 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xlffx"] Jul 10 18:29:43.009515 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:43.009461 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:43.009578 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:43.009540 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:43.018230 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:43.018190 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" podStartSLOduration=10.244530671 podStartE2EDuration="29.018178045s" totalImagesPullingTime="18.773647374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:43.017851971 +0000 UTC m=+29.695564432" watchObservedRunningTime="2026-07-10 18:29:43.018178045 +0000 UTC m=+29.695890485" Jul 10 18:29:43.983591 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:43.983524 2576 generic.go:356] "Generic (PLEG): container finished" podID="91835427-78a0-464d-87ab-d70b45a77388" containerID="ad6ebce778d6e03448d1c174c24586f38151ff405cb459d05a9ba3dd3993a2c0" exitCode=0 Jul 10 18:29:43.984020 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:43.983611 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94n7z" event={"ID":"91835427-78a0-464d-87ab-d70b45a77388","Type":"ContainerDied","Data":"ad6ebce778d6e03448d1c174c24586f38151ff405cb459d05a9ba3dd3993a2c0"} Jul 10 18:29:44.838190 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:44.838159 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:44.838334 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:44.838159 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:44.838334 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:44.838265 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:44.838433 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:44.838339 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:44.838433 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:44.838160 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:44.838433 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:44.838424 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:45.989669 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:45.989481 2576 generic.go:356] "Generic (PLEG): container finished" podID="91835427-78a0-464d-87ab-d70b45a77388" containerID="fafe91b64e26867cb714ed9b3f4c643e2b307a593fc5f254729f1aa85c1a5830" exitCode=0 Jul 10 18:29:45.989998 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:45.989567 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94n7z" event={"ID":"91835427-78a0-464d-87ab-d70b45a77388","Type":"ContainerDied","Data":"fafe91b64e26867cb714ed9b3f4c643e2b307a593fc5f254729f1aa85c1a5830"} Jul 10 18:29:46.837836 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:46.837785 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:46.838024 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:46.837802 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:46.838024 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:46.837898 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqrqz" podUID="74645fb8-cd43-4907-b0dc-3446ed74822b" Jul 10 18:29:46.838024 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:46.837913 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:46.838024 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:46.837970 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xlffx" podUID="43310758-62c8-4c7b-8c6b-4fe08aa348d1" Jul 10 18:29:46.838188 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:46.838054 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs2hc" podUID="ec189118-e07f-4ba5-a5c0-cc1910849ca6" Jul 10 18:29:47.506599 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:47.506558 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs\") pod \"network-metrics-daemon-xlffx\" (UID: \"43310758-62c8-4c7b-8c6b-4fe08aa348d1\") " pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:47.506599 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:47.506599 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:47.507107 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:47.506721 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:29:47.507107 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:47.506722 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:29:47.507107 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:47.506792 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret podName:ec189118-e07f-4ba5-a5c0-cc1910849ca6 nodeName:}" failed. No retries permitted until 2026-07-10 18:30:19.506773428 +0000 UTC m=+66.184485868 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret") pod "global-pull-secret-syncer-rs2hc" (UID: "ec189118-e07f-4ba5-a5c0-cc1910849ca6") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:29:47.507107 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:47.506813 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs podName:43310758-62c8-4c7b-8c6b-4fe08aa348d1 nodeName:}" failed. No retries permitted until 2026-07-10 18:30:19.506804516 +0000 UTC m=+66.184516956 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs") pod "network-metrics-daemon-xlffx" (UID: "43310758-62c8-4c7b-8c6b-4fe08aa348d1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:29:47.607619 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:47.607577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwz9\" (UniqueName: \"kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9\") pod \"network-check-target-lqrqz\" (UID: \"74645fb8-cd43-4907-b0dc-3446ed74822b\") " pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:47.607762 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:47.607746 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:29:47.607811 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:47.607765 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:29:47.607811 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:47.607779 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ftwz9 for pod openshift-network-diagnostics/network-check-target-lqrqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:29:47.607917 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:47.607835 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9 podName:74645fb8-cd43-4907-b0dc-3446ed74822b nodeName:}" failed. No retries permitted until 2026-07-10 18:30:19.607815726 +0000 UTC m=+66.285528167 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ftwz9" (UniqueName: "kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9") pod "network-check-target-lqrqz" (UID: "74645fb8-cd43-4907-b0dc-3446ed74822b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:29:48.097393 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.097346 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-180.ec2.internal" event="NodeReady" Jul 10 18:29:48.097564 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.097524 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 18:29:48.152770 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.152743 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847c599756-654rz"] Jul 10 18:29:48.156169 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.156146 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847c599756-654rz" Jul 10 18:29:48.160097 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.160069 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q"] Jul 10 18:29:48.160271 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.160251 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 10 18:29:48.161331 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.160710 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4z8fc\"" Jul 10 18:29:48.161331 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.160736 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 18:29:48.161331 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.161057 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 18:29:48.162840 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.162819 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 18:29:48.163866 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.163847 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df"] Jul 10 18:29:48.163988 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.163968 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" Jul 10 18:29:48.166655 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.166636 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 10 18:29:48.167847 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.167828 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb"] Jul 10 18:29:48.167973 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.167958 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.171274 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.171248 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.172470 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.172443 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 18:29:48.172470 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.172453 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 18:29:48.172623 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.172528 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 18:29:48.173313 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.173295 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847c599756-654rz"] Jul 10 18:29:48.174807 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.174788 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 18:29:48.177378 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.177276 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 18:29:48.178120 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.177959 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6l98j\"" Jul 10 18:29:48.178275 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.178258 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 18:29:48.178400 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.178380 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 18:29:48.179330 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.179282 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q"] Jul 10 18:29:48.186870 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.186851 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 18:29:48.188498 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.188474 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb"] Jul 10 18:29:48.190845 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.190818 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qlqq4"] Jul 10 18:29:48.195024 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.195005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:48.199637 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.199531 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 18:29:48.200001 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.199867 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 18:29:48.200001 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.199867 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hf98r\"" Jul 10 18:29:48.200823 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.200804 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df"] Jul 10 18:29:48.210610 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.210588 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qlqq4"] Jul 10 18:29:48.273943 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.273914 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-75cm8"] Jul 10 18:29:48.277747 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.277721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-75cm8" Jul 10 18:29:48.280548 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.280301 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ccz8c\"" Jul 10 18:29:48.280548 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.280312 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 18:29:48.280548 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.280326 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 18:29:48.280548 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.280474 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 18:29:48.288383 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.288344 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-75cm8"] Jul 10 18:29:48.311284 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55555f8f-ae9d-48eb-897f-f1fcc223a5da-trusted-ca\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.311475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311300 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8h8m\" (UniqueName: \"kubernetes.io/projected/dfd2ef24-efdf-4943-a435-8c4f1cca5789-kube-api-access-c8h8m\") pod \"klusterlet-addon-workmgr-554c889f55-gg87q\" (UID: \"dfd2ef24-efdf-4943-a435-8c4f1cca5789\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" Jul 10 18:29:48.311475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55555f8f-ae9d-48eb-897f-f1fcc223a5da-image-registry-private-configuration\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.311475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/870983c6-1d62-4b03-9553-6eadcca60db8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-847c599756-654rz\" (UID: \"870983c6-1d62-4b03-9553-6eadcca60db8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847c599756-654rz" Jul 10 18:29:48.311475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8hd9\" (UniqueName: \"kubernetes.io/projected/870983c6-1d62-4b03-9553-6eadcca60db8-kube-api-access-j8hd9\") pod \"managed-serviceaccount-addon-agent-847c599756-654rz\" (UID: \"870983c6-1d62-4b03-9553-6eadcca60db8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847c599756-654rz" Jul 10 18:29:48.311475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311420 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6b89a502-aa73-4b53-a31d-461bcde246f5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.311475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwj49\" (UniqueName: \"kubernetes.io/projected/6b89a502-aa73-4b53-a31d-461bcde246f5-kube-api-access-rwj49\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.311475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311456 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6b89a502-aa73-4b53-a31d-461bcde246f5-hub\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.311836 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55555f8f-ae9d-48eb-897f-f1fcc223a5da-ca-trust-extracted\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.311836 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfd2ef24-efdf-4943-a435-8c4f1cca5789-tmp\") pod \"klusterlet-addon-workmgr-554c889f55-gg87q\" (UID: \"dfd2ef24-efdf-4943-a435-8c4f1cca5789\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" Jul 10 18:29:48.311836 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311558 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-bound-sa-token\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.311836 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b89a502-aa73-4b53-a31d-461bcde246f5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.311836 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311613 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxn9m\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-kube-api-access-hxn9m\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.311836 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-certificates\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.311836 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311755 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55555f8f-ae9d-48eb-897f-f1fcc223a5da-installation-pull-secrets\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.311836 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.311836 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:48.312161 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6b89a502-aa73-4b53-a31d-461bcde246f5-ca\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.312161 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fca207e-d050-4f78-beee-b9f61906bf18-config-volume\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:48.312161 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fca207e-d050-4f78-beee-b9f61906bf18-tmp-dir\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:48.312161 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjdtb\" (UniqueName: \"kubernetes.io/projected/4fca207e-d050-4f78-beee-b9f61906bf18-kube-api-access-xjdtb\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:48.312161 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dfd2ef24-efdf-4943-a435-8c4f1cca5789-klusterlet-config\") pod \"klusterlet-addon-workmgr-554c889f55-gg87q\" (UID: \"dfd2ef24-efdf-4943-a435-8c4f1cca5789\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" Jul 10 18:29:48.312161 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.311998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b89a502-aa73-4b53-a31d-461bcde246f5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.413171 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6b89a502-aa73-4b53-a31d-461bcde246f5-hub\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.413171 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413131 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55555f8f-ae9d-48eb-897f-f1fcc223a5da-ca-trust-extracted\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.413400 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413283 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfd2ef24-efdf-4943-a435-8c4f1cca5789-tmp\") pod \"klusterlet-addon-workmgr-554c889f55-gg87q\" (UID: \"dfd2ef24-efdf-4943-a435-8c4f1cca5789\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" Jul 10 18:29:48.413400 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-bound-sa-token\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.413400 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413361 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5ff0e60-3cce-449d-ad20-f872820b48c2-cert\") pod \"ingress-canary-75cm8\" (UID: \"f5ff0e60-3cce-449d-ad20-f872820b48c2\") " pod="openshift-ingress-canary/ingress-canary-75cm8" Jul 10 18:29:48.413555 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413417 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b89a502-aa73-4b53-a31d-461bcde246f5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.413555 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxn9m\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-kube-api-access-hxn9m\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.413555 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9glxk\" (UniqueName: \"kubernetes.io/projected/f5ff0e60-3cce-449d-ad20-f872820b48c2-kube-api-access-9glxk\") pod \"ingress-canary-75cm8\" (UID: \"f5ff0e60-3cce-449d-ad20-f872820b48c2\") " pod="openshift-ingress-canary/ingress-canary-75cm8" Jul 10 18:29:48.413555 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-certificates\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.413555 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55555f8f-ae9d-48eb-897f-f1fcc223a5da-installation-pull-secrets\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.413789 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.413789 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:48.413789 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413618 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6b89a502-aa73-4b53-a31d-461bcde246f5-ca\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.413789 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fca207e-d050-4f78-beee-b9f61906bf18-config-volume\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:48.413789 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413666 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55555f8f-ae9d-48eb-897f-f1fcc223a5da-ca-trust-extracted\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.413789 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fca207e-d050-4f78-beee-b9f61906bf18-tmp-dir\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:48.413789 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjdtb\" (UniqueName: \"kubernetes.io/projected/4fca207e-d050-4f78-beee-b9f61906bf18-kube-api-access-xjdtb\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:48.413789 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dfd2ef24-efdf-4943-a435-8c4f1cca5789-klusterlet-config\") pod \"klusterlet-addon-workmgr-554c889f55-gg87q\" (UID: \"dfd2ef24-efdf-4943-a435-8c4f1cca5789\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" Jul 10 18:29:48.413789 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413779 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b89a502-aa73-4b53-a31d-461bcde246f5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.414175 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55555f8f-ae9d-48eb-897f-f1fcc223a5da-trusted-ca\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.414175 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8h8m\" (UniqueName: \"kubernetes.io/projected/dfd2ef24-efdf-4943-a435-8c4f1cca5789-kube-api-access-c8h8m\") pod \"klusterlet-addon-workmgr-554c889f55-gg87q\" (UID: \"dfd2ef24-efdf-4943-a435-8c4f1cca5789\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" Jul 10 18:29:48.414175 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55555f8f-ae9d-48eb-897f-f1fcc223a5da-image-registry-private-configuration\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.414175 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/870983c6-1d62-4b03-9553-6eadcca60db8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-847c599756-654rz\" (UID: \"870983c6-1d62-4b03-9553-6eadcca60db8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847c599756-654rz" Jul 10 18:29:48.414175 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.413988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8hd9\" (UniqueName: \"kubernetes.io/projected/870983c6-1d62-4b03-9553-6eadcca60db8-kube-api-access-j8hd9\") pod \"managed-serviceaccount-addon-agent-847c599756-654rz\" (UID: \"870983c6-1d62-4b03-9553-6eadcca60db8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847c599756-654rz" Jul 10 18:29:48.414175 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.414054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6b89a502-aa73-4b53-a31d-461bcde246f5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.414175 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.414075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwj49\" (UniqueName: \"kubernetes.io/projected/6b89a502-aa73-4b53-a31d-461bcde246f5-kube-api-access-rwj49\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.414175 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.414146 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-certificates\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.414581 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:48.414267 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:29:48.414581 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:48.414285 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb: secret "image-registry-tls" not found Jul 10 18:29:48.414581 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:48.414369 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls podName:55555f8f-ae9d-48eb-897f-f1fcc223a5da nodeName:}" failed. No retries permitted until 2026-07-10 18:29:48.914327058 +0000 UTC m=+35.592039511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls") pod "image-registry-6bdbf8d7f4-ztxbb" (UID: "55555f8f-ae9d-48eb-897f-f1fcc223a5da") : secret "image-registry-tls" not found Jul 10 18:29:48.414744 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.414713 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfd2ef24-efdf-4943-a435-8c4f1cca5789-tmp\") pod \"klusterlet-addon-workmgr-554c889f55-gg87q\" (UID: \"dfd2ef24-efdf-4943-a435-8c4f1cca5789\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" Jul 10 18:29:48.415298 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:48.414903 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:29:48.415298 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:48.414970 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls podName:4fca207e-d050-4f78-beee-b9f61906bf18 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:48.914951832 +0000 UTC m=+35.592664294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls") pod "dns-default-qlqq4" (UID: "4fca207e-d050-4f78-beee-b9f61906bf18") : secret "dns-default-metrics-tls" not found Jul 10 18:29:48.415298 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.415057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fca207e-d050-4f78-beee-b9f61906bf18-config-volume\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:48.415543 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.415503 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fca207e-d050-4f78-beee-b9f61906bf18-tmp-dir\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:48.416172 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.416140 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b89a502-aa73-4b53-a31d-461bcde246f5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.417151 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.416846 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55555f8f-ae9d-48eb-897f-f1fcc223a5da-trusted-ca\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.418720 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.418523 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55555f8f-ae9d-48eb-897f-f1fcc223a5da-image-registry-private-configuration\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.418720 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.418533 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6b89a502-aa73-4b53-a31d-461bcde246f5-hub\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.418720 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.418547 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dfd2ef24-efdf-4943-a435-8c4f1cca5789-klusterlet-config\") pod \"klusterlet-addon-workmgr-554c889f55-gg87q\" (UID: \"dfd2ef24-efdf-4943-a435-8c4f1cca5789\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" Jul 10 18:29:48.418926 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.418897 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6b89a502-aa73-4b53-a31d-461bcde246f5-ca\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.419775 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.419751 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b89a502-aa73-4b53-a31d-461bcde246f5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.419875 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.419789 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6b89a502-aa73-4b53-a31d-461bcde246f5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.420126 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.420107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55555f8f-ae9d-48eb-897f-f1fcc223a5da-installation-pull-secrets\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.421032 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.421013 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/870983c6-1d62-4b03-9553-6eadcca60db8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-847c599756-654rz\" (UID: \"870983c6-1d62-4b03-9553-6eadcca60db8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847c599756-654rz" Jul 10 18:29:48.425299 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.425255 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwj49\" (UniqueName: \"kubernetes.io/projected/6b89a502-aa73-4b53-a31d-461bcde246f5-kube-api-access-rwj49\") pod \"cluster-proxy-proxy-agent-f7b7d9cf-8w9df\" (UID: \"6b89a502-aa73-4b53-a31d-461bcde246f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.425299 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.425278 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8h8m\" (UniqueName: \"kubernetes.io/projected/dfd2ef24-efdf-4943-a435-8c4f1cca5789-kube-api-access-c8h8m\") pod \"klusterlet-addon-workmgr-554c889f55-gg87q\" (UID: \"dfd2ef24-efdf-4943-a435-8c4f1cca5789\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" Jul 10 18:29:48.426754 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.426728 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8hd9\" (UniqueName: \"kubernetes.io/projected/870983c6-1d62-4b03-9553-6eadcca60db8-kube-api-access-j8hd9\") pod \"managed-serviceaccount-addon-agent-847c599756-654rz\" (UID: \"870983c6-1d62-4b03-9553-6eadcca60db8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847c599756-654rz" Jul 10 18:29:48.426840 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.426764 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjdtb\" (UniqueName: \"kubernetes.io/projected/4fca207e-d050-4f78-beee-b9f61906bf18-kube-api-access-xjdtb\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:48.428098 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.427779 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-bound-sa-token\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.428098 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.427978 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxn9m\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-kube-api-access-hxn9m\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.476911 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.476869 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847c599756-654rz" Jul 10 18:29:48.486490 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.486458 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" Jul 10 18:29:48.496137 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.496104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:29:48.516199 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.515422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5ff0e60-3cce-449d-ad20-f872820b48c2-cert\") pod \"ingress-canary-75cm8\" (UID: \"f5ff0e60-3cce-449d-ad20-f872820b48c2\") " pod="openshift-ingress-canary/ingress-canary-75cm8" Jul 10 18:29:48.516199 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.515492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9glxk\" (UniqueName: \"kubernetes.io/projected/f5ff0e60-3cce-449d-ad20-f872820b48c2-kube-api-access-9glxk\") pod \"ingress-canary-75cm8\" (UID: \"f5ff0e60-3cce-449d-ad20-f872820b48c2\") " pod="openshift-ingress-canary/ingress-canary-75cm8" Jul 10 18:29:48.518986 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.518963 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5ff0e60-3cce-449d-ad20-f872820b48c2-cert\") pod \"ingress-canary-75cm8\" (UID: \"f5ff0e60-3cce-449d-ad20-f872820b48c2\") " pod="openshift-ingress-canary/ingress-canary-75cm8" Jul 10 18:29:48.525727 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.525674 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9glxk\" (UniqueName: \"kubernetes.io/projected/f5ff0e60-3cce-449d-ad20-f872820b48c2-kube-api-access-9glxk\") pod \"ingress-canary-75cm8\" (UID: \"f5ff0e60-3cce-449d-ad20-f872820b48c2\") " pod="openshift-ingress-canary/ingress-canary-75cm8" Jul 10 18:29:48.588220 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.588184 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-75cm8" Jul 10 18:29:48.650157 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.649789 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847c599756-654rz"] Jul 10 18:29:48.657713 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:48.657648 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod870983c6_1d62_4b03_9553_6eadcca60db8.slice/crio-d4f860fba63f8c173cb2b99bb25ba865d419504f7f6590a1d2523d385c2bc491 WatchSource:0}: Error finding container d4f860fba63f8c173cb2b99bb25ba865d419504f7f6590a1d2523d385c2bc491: Status 404 returned error can't find the container with id d4f860fba63f8c173cb2b99bb25ba865d419504f7f6590a1d2523d385c2bc491 Jul 10 18:29:48.660891 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.660381 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q"] Jul 10 18:29:48.663520 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:48.663437 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd2ef24_efdf_4943_a435_8c4f1cca5789.slice/crio-a6bd5573c040ab07fae08cdb832c7c5050dabf664a7f03abcf94cf08e5aa396f WatchSource:0}: Error finding container a6bd5573c040ab07fae08cdb832c7c5050dabf664a7f03abcf94cf08e5aa396f: Status 404 returned error can't find the container with id a6bd5573c040ab07fae08cdb832c7c5050dabf664a7f03abcf94cf08e5aa396f Jul 10 18:29:48.668283 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.668236 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df"] Jul 10 18:29:48.672269 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:48.672244 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b89a502_aa73_4b53_a31d_461bcde246f5.slice/crio-50d5e10aa927977b3ae693669fae3e487ac9317e65638af476f9213daa072b8c WatchSource:0}: Error finding container 50d5e10aa927977b3ae693669fae3e487ac9317e65638af476f9213daa072b8c: Status 404 returned error can't find the container with id 50d5e10aa927977b3ae693669fae3e487ac9317e65638af476f9213daa072b8c Jul 10 18:29:48.741285 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.741259 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-75cm8"] Jul 10 18:29:48.744500 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:29:48.744469 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ff0e60_3cce_449d_ad20_f872820b48c2.slice/crio-d690a5d619a897cd27135716facc030c16c4b5faae4425f6bc02b003b8a209c8 WatchSource:0}: Error finding container d690a5d619a897cd27135716facc030c16c4b5faae4425f6bc02b003b8a209c8: Status 404 returned error can't find the container with id d690a5d619a897cd27135716facc030c16c4b5faae4425f6bc02b003b8a209c8 Jul 10 18:29:48.838061 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.838027 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:29:48.838061 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.838056 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:29:48.838237 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.838032 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:29:48.843702 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.841432 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 18:29:48.843702 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.843639 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hhh94\"" Jul 10 18:29:48.843702 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.843673 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 18:29:48.843956 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.843741 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 18:29:48.843956 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.843677 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 18:29:48.845090 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.844773 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tzdtl\"" Jul 10 18:29:48.923585 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.923515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:48.923585 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.923549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:48.923753 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:48.923675 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:29:48.923753 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:48.923680 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:29:48.923753 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:48.923694 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb: secret "image-registry-tls" not found Jul 10 18:29:48.923753 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:48.923736 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls podName:4fca207e-d050-4f78-beee-b9f61906bf18 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:49.923718321 +0000 UTC m=+36.601430767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls") pod "dns-default-qlqq4" (UID: "4fca207e-d050-4f78-beee-b9f61906bf18") : secret "dns-default-metrics-tls" not found Jul 10 18:29:48.923753 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:48.923749 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls podName:55555f8f-ae9d-48eb-897f-f1fcc223a5da nodeName:}" failed. No retries permitted until 2026-07-10 18:29:49.923744101 +0000 UTC m=+36.601456540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls") pod "image-registry-6bdbf8d7f4-ztxbb" (UID: "55555f8f-ae9d-48eb-897f-f1fcc223a5da") : secret "image-registry-tls" not found Jul 10 18:29:48.998497 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.998463 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" event={"ID":"dfd2ef24-efdf-4943-a435-8c4f1cca5789","Type":"ContainerStarted","Data":"a6bd5573c040ab07fae08cdb832c7c5050dabf664a7f03abcf94cf08e5aa396f"} Jul 10 18:29:48.999912 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:48.999878 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847c599756-654rz" event={"ID":"870983c6-1d62-4b03-9553-6eadcca60db8","Type":"ContainerStarted","Data":"d4f860fba63f8c173cb2b99bb25ba865d419504f7f6590a1d2523d385c2bc491"} Jul 10 18:29:49.001162 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:49.001125 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" event={"ID":"6b89a502-aa73-4b53-a31d-461bcde246f5","Type":"ContainerStarted","Data":"50d5e10aa927977b3ae693669fae3e487ac9317e65638af476f9213daa072b8c"} Jul 10 18:29:49.002297 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:49.002276 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-75cm8" event={"ID":"f5ff0e60-3cce-449d-ad20-f872820b48c2","Type":"ContainerStarted","Data":"d690a5d619a897cd27135716facc030c16c4b5faae4425f6bc02b003b8a209c8"} Jul 10 18:29:49.932060 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:49.932026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:49.932060 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:49.932068 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:49.932682 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:49.932204 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:29:49.932682 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:49.932231 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb: secret "image-registry-tls" not found Jul 10 18:29:49.932682 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:49.932289 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls podName:55555f8f-ae9d-48eb-897f-f1fcc223a5da nodeName:}" failed. No retries permitted until 2026-07-10 18:29:51.932274214 +0000 UTC m=+38.609986654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls") pod "image-registry-6bdbf8d7f4-ztxbb" (UID: "55555f8f-ae9d-48eb-897f-f1fcc223a5da") : secret "image-registry-tls" not found Jul 10 18:29:49.932682 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:49.932207 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:29:49.932682 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:49.932368 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls podName:4fca207e-d050-4f78-beee-b9f61906bf18 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:51.93233797 +0000 UTC m=+38.610050412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls") pod "dns-default-qlqq4" (UID: "4fca207e-d050-4f78-beee-b9f61906bf18") : secret "dns-default-metrics-tls" not found Jul 10 18:29:51.023340 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:51.023301 2576 generic.go:356] "Generic (PLEG): container finished" podID="91835427-78a0-464d-87ab-d70b45a77388" containerID="8d5c7a8566103ededc63b98762ef9175fc86c818d37e41a5466750ca83277a5a" exitCode=0 Jul 10 18:29:51.023782 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:51.023390 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94n7z" event={"ID":"91835427-78a0-464d-87ab-d70b45a77388","Type":"ContainerDied","Data":"8d5c7a8566103ededc63b98762ef9175fc86c818d37e41a5466750ca83277a5a"} Jul 10 18:29:51.950458 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:51.950419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:51.950632 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:51.950466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:51.950691 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:51.950616 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:29:51.950753 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:51.950702 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls podName:4fca207e-d050-4f78-beee-b9f61906bf18 nodeName:}" failed. No retries permitted until 2026-07-10 18:29:55.950680824 +0000 UTC m=+42.628393282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls") pod "dns-default-qlqq4" (UID: "4fca207e-d050-4f78-beee-b9f61906bf18") : secret "dns-default-metrics-tls" not found Jul 10 18:29:51.951363 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:51.951157 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:29:51.951363 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:51.951177 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb: secret "image-registry-tls" not found Jul 10 18:29:51.951363 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:51.951227 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls podName:55555f8f-ae9d-48eb-897f-f1fcc223a5da nodeName:}" failed. No retries permitted until 2026-07-10 18:29:55.951207335 +0000 UTC m=+42.628919776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls") pod "image-registry-6bdbf8d7f4-ztxbb" (UID: "55555f8f-ae9d-48eb-897f-f1fcc223a5da") : secret "image-registry-tls" not found Jul 10 18:29:52.037515 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:52.036760 2576 generic.go:356] "Generic (PLEG): container finished" podID="91835427-78a0-464d-87ab-d70b45a77388" containerID="cd519cfaf5a892e0a3e060651bdfe42888293f2713ca145400d1c5d8f2638f47" exitCode=0 Jul 10 18:29:52.037515 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:52.036816 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94n7z" event={"ID":"91835427-78a0-464d-87ab-d70b45a77388","Type":"ContainerDied","Data":"cd519cfaf5a892e0a3e060651bdfe42888293f2713ca145400d1c5d8f2638f47"} Jul 10 18:29:53.046458 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:53.046119 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94n7z" event={"ID":"91835427-78a0-464d-87ab-d70b45a77388","Type":"ContainerStarted","Data":"9cf23843714f10f61841af1a1fd0dc046aed6f9d56562845dd1cb338a16560ab"} Jul 10 18:29:53.084725 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:53.083675 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-94n7z" podStartSLOduration=8.239772904 podStartE2EDuration="39.083647074s" totalImagesPullingTime="30.84387417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:29:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:53.081894971 +0000 UTC m=+39.759607444" watchObservedRunningTime="2026-07-10 18:29:53.083647074 +0000 UTC m=+39.761359539" Jul 10 18:29:55.987314 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:55.987277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:29:55.987314 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:55.987314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:29:55.987761 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:55.987445 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:29:55.987761 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:55.987497 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls podName:4fca207e-d050-4f78-beee-b9f61906bf18 nodeName:}" failed. No retries permitted until 2026-07-10 18:30:03.987483767 +0000 UTC m=+50.665196207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls") pod "dns-default-qlqq4" (UID: "4fca207e-d050-4f78-beee-b9f61906bf18") : secret "dns-default-metrics-tls" not found Jul 10 18:29:55.987761 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:55.987448 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:29:55.987761 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:55.987518 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb: secret "image-registry-tls" not found Jul 10 18:29:55.987761 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:29:55.987553 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls podName:55555f8f-ae9d-48eb-897f-f1fcc223a5da nodeName:}" failed. No retries permitted until 2026-07-10 18:30:03.987542906 +0000 UTC m=+50.665255360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls") pod "image-registry-6bdbf8d7f4-ztxbb" (UID: "55555f8f-ae9d-48eb-897f-f1fcc223a5da") : secret "image-registry-tls" not found Jul 10 18:29:59.062914 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:59.062878 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847c599756-654rz" event={"ID":"870983c6-1d62-4b03-9553-6eadcca60db8","Type":"ContainerStarted","Data":"4458a804b4b07ad73c05702624a443aed8b608723a0d21936f51bd03760a3e91"} Jul 10 18:29:59.064836 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:59.064807 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" event={"ID":"6b89a502-aa73-4b53-a31d-461bcde246f5","Type":"ContainerStarted","Data":"ff1048e908d40da28925d99828ad594f76ad639fd1f656e56bbbe7ea7e0947f6"} Jul 10 18:29:59.064944 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:59.064845 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" event={"ID":"6b89a502-aa73-4b53-a31d-461bcde246f5","Type":"ContainerStarted","Data":"8a231b43e7172e8530a831f82ee54eb969c0830dfa1ffb643e9e62a5fdb42d1e"} Jul 10 18:29:59.064944 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:59.064867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" event={"ID":"6b89a502-aa73-4b53-a31d-461bcde246f5","Type":"ContainerStarted","Data":"906cdbb03fe08088e75302aa4c8c6dcb783368b0b17dd5fd65e93a5b6dc8bc38"} Jul 10 18:29:59.066086 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:59.066067 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-75cm8" event={"ID":"f5ff0e60-3cce-449d-ad20-f872820b48c2","Type":"ContainerStarted","Data":"7db2233cbeb1b898b668583ebc33042d781bcd1141a45f39a8b7ea846ebf2209"} Jul 10 18:29:59.067496 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:59.067475 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" event={"ID":"dfd2ef24-efdf-4943-a435-8c4f1cca5789","Type":"ContainerStarted","Data":"73025d1191d973b49fe0b76042aa107ba4930e19b69cdb9ef5d356e1285b059b"} Jul 10 18:29:59.067689 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:59.067655 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" Jul 10 18:29:59.069314 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:59.069294 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" Jul 10 18:29:59.082116 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:59.082064 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-847c599756-654rz" podStartSLOduration=16.534755888 podStartE2EDuration="26.082049047s" totalImagesPullingTime="9.547293159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:59.081634821 +0000 UTC m=+45.759347283" watchObservedRunningTime="2026-07-10 18:29:59.082049047 +0000 UTC m=+45.759761512" Jul 10 18:29:59.120479 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:59.120430 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-554c889f55-gg87q" podStartSLOduration=16.576804053 podStartE2EDuration="26.12041683s" totalImagesPullingTime="9.543612777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:59.100070253 +0000 UTC m=+45.777782714" watchObservedRunningTime="2026-07-10 18:29:59.12041683 +0000 UTC m=+45.798129271" Jul 10 18:29:59.120618 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:59.120530 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" podStartSLOduration=16.569958675 podStartE2EDuration="26.12051899s" totalImagesPullingTime="9.550560315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:59.119645722 +0000 UTC m=+45.797358183" watchObservedRunningTime="2026-07-10 18:29:59.12051899 +0000 UTC m=+45.798231453" Jul 10 18:29:59.134239 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:29:59.134192 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-75cm8" podStartSLOduration=1.6717650910000001 podStartE2EDuration="11.134176052s" totalImagesPullingTime="9.462410961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:29:59.134072699 +0000 UTC m=+45.811785164" watchObservedRunningTime="2026-07-10 18:29:59.134176052 +0000 UTC m=+45.811888515" Jul 10 18:30:04.047325 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:04.047283 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:30:04.047325 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:04.047324 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:30:04.047790 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:04.047439 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:30:04.047790 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:04.047443 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:30:04.047790 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:04.047461 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb: secret "image-registry-tls" not found Jul 10 18:30:04.047790 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:04.047489 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls podName:4fca207e-d050-4f78-beee-b9f61906bf18 nodeName:}" failed. No retries permitted until 2026-07-10 18:30:20.047476729 +0000 UTC m=+66.725189169 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls") pod "dns-default-qlqq4" (UID: "4fca207e-d050-4f78-beee-b9f61906bf18") : secret "dns-default-metrics-tls" not found Jul 10 18:30:04.047790 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:04.047501 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls podName:55555f8f-ae9d-48eb-897f-f1fcc223a5da nodeName:}" failed. No retries permitted until 2026-07-10 18:30:20.047495591 +0000 UTC m=+66.725208031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls") pod "image-registry-6bdbf8d7f4-ztxbb" (UID: "55555f8f-ae9d-48eb-897f-f1fcc223a5da") : secret "image-registry-tls" not found Jul 10 18:30:14.996505 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:14.996479 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzct8" Jul 10 18:30:19.192532 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.192494 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nrqp5"] Jul 10 18:30:19.195127 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.195109 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.198121 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.198098 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 18:30:19.198864 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.198842 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-54lkq\"" Jul 10 18:30:19.198980 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.198886 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 18:30:19.198980 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.198900 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 18:30:19.198980 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.198890 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 18:30:19.212049 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.212028 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nrqp5"] Jul 10 18:30:19.262557 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.262517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12d0fea4-abb3-467a-8422-7e1a81d723b3-data-volume\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.262557 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.262557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpkmc\" (UniqueName: \"kubernetes.io/projected/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-api-access-tpkmc\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.262751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.262587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12d0fea4-abb3-467a-8422-7e1a81d723b3-crio-socket\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.262751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.262649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.262751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.262735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.363870 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.363840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12d0fea4-abb3-467a-8422-7e1a81d723b3-data-volume\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.363974 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.363876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpkmc\" (UniqueName: \"kubernetes.io/projected/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-api-access-tpkmc\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.363974 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.363911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12d0fea4-abb3-467a-8422-7e1a81d723b3-crio-socket\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.363974 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.363934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.364076 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.363996 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.364076 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.364054 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12d0fea4-abb3-467a-8422-7e1a81d723b3-crio-socket\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.364140 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:19.364069 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:30:19.364140 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:19.364079 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:30:19.364140 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:19.364127 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls podName:12d0fea4-abb3-467a-8422-7e1a81d723b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:30:19.864114448 +0000 UTC m=+66.541826889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nrqp5" (UID: "12d0fea4-abb3-467a-8422-7e1a81d723b3") : secret "insights-runtime-extractor-tls" not found Jul 10 18:30:19.364242 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:19.364150 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm podName:12d0fea4-abb3-467a-8422-7e1a81d723b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:30:19.864131908 +0000 UTC m=+66.541844370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nrqp5" (UID: "12d0fea4-abb3-467a-8422-7e1a81d723b3") : configmap "kube-rbac-proxy" not found Jul 10 18:30:19.364242 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.364210 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12d0fea4-abb3-467a-8422-7e1a81d723b3-data-volume\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.374189 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.374168 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpkmc\" (UniqueName: \"kubernetes.io/projected/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-api-access-tpkmc\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.565479 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.565448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:30:19.565631 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.565520 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs\") pod \"network-metrics-daemon-xlffx\" (UID: \"43310758-62c8-4c7b-8c6b-4fe08aa348d1\") " pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:30:19.568111 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.568091 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 18:30:19.568218 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.568097 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 18:30:19.576169 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:19.576152 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 18:30:19.576233 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:19.576212 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs podName:43310758-62c8-4c7b-8c6b-4fe08aa348d1 nodeName:}" failed. No retries permitted until 2026-07-10 18:31:23.576198734 +0000 UTC m=+130.253911174 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs") pod "network-metrics-daemon-xlffx" (UID: "43310758-62c8-4c7b-8c6b-4fe08aa348d1") : secret "metrics-daemon-secret" not found Jul 10 18:30:19.578153 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.578135 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec189118-e07f-4ba5-a5c0-cc1910849ca6-original-pull-secret\") pod \"global-pull-secret-syncer-rs2hc\" (UID: \"ec189118-e07f-4ba5-a5c0-cc1910849ca6\") " pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:30:19.666655 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.666619 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwz9\" (UniqueName: \"kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9\") pod \"network-check-target-lqrqz\" (UID: \"74645fb8-cd43-4907-b0dc-3446ed74822b\") " pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:30:19.669121 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.669101 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 18:30:19.679240 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.679222 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 18:30:19.690481 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.690456 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftwz9\" (UniqueName: \"kubernetes.io/projected/74645fb8-cd43-4907-b0dc-3446ed74822b-kube-api-access-ftwz9\") pod \"network-check-target-lqrqz\" (UID: \"74645fb8-cd43-4907-b0dc-3446ed74822b\") " pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:30:19.758072 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.758042 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tzdtl\"" Jul 10 18:30:19.766063 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.766042 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:30:19.769831 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.769808 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs2hc" Jul 10 18:30:19.867881 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.867849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.868044 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.867963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:19.868044 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:19.868004 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:30:19.868154 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:19.868058 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:30:19.868154 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:19.868066 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls podName:12d0fea4-abb3-467a-8422-7e1a81d723b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:30:20.868050807 +0000 UTC m=+67.545763249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nrqp5" (UID: "12d0fea4-abb3-467a-8422-7e1a81d723b3") : secret "insights-runtime-extractor-tls" not found Jul 10 18:30:19.868154 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:19.868110 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm podName:12d0fea4-abb3-467a-8422-7e1a81d723b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:30:20.868094546 +0000 UTC m=+67.545806992 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nrqp5" (UID: "12d0fea4-abb3-467a-8422-7e1a81d723b3") : configmap "kube-rbac-proxy" not found Jul 10 18:30:19.898454 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.898346 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lqrqz"] Jul 10 18:30:19.901583 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:30:19.901557 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74645fb8_cd43_4907_b0dc_3446ed74822b.slice/crio-a6fb0594cc84bf08458b5d5109c9763726301a3f72126b953cbcccfc42689255 WatchSource:0}: Error finding container a6fb0594cc84bf08458b5d5109c9763726301a3f72126b953cbcccfc42689255: Status 404 returned error can't find the container with id a6fb0594cc84bf08458b5d5109c9763726301a3f72126b953cbcccfc42689255 Jul 10 18:30:19.916714 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:19.916687 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rs2hc"] Jul 10 18:30:19.919318 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:30:19.919297 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec189118_e07f_4ba5_a5c0_cc1910849ca6.slice/crio-1c7313549db70639834ad55bed32bda3ca4b4c14c5e32635e757d8ca48dc0f43 WatchSource:0}: Error finding container 1c7313549db70639834ad55bed32bda3ca4b4c14c5e32635e757d8ca48dc0f43: Status 404 returned error can't find the container with id 1c7313549db70639834ad55bed32bda3ca4b4c14c5e32635e757d8ca48dc0f43 Jul 10 18:30:20.069494 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:20.069410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:30:20.069628 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:20.069497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:30:20.069628 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:20.069553 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:30:20.069628 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:20.069570 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb: secret "image-registry-tls" not found Jul 10 18:30:20.069628 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:20.069593 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:30:20.069749 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:20.069630 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls podName:55555f8f-ae9d-48eb-897f-f1fcc223a5da nodeName:}" failed. No retries permitted until 2026-07-10 18:30:52.069613925 +0000 UTC m=+98.747326368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls") pod "image-registry-6bdbf8d7f4-ztxbb" (UID: "55555f8f-ae9d-48eb-897f-f1fcc223a5da") : secret "image-registry-tls" not found Jul 10 18:30:20.069749 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:20.069643 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls podName:4fca207e-d050-4f78-beee-b9f61906bf18 nodeName:}" failed. No retries permitted until 2026-07-10 18:30:52.069637077 +0000 UTC m=+98.747349517 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls") pod "dns-default-qlqq4" (UID: "4fca207e-d050-4f78-beee-b9f61906bf18") : secret "dns-default-metrics-tls" not found Jul 10 18:30:20.127688 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:20.127644 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rs2hc" event={"ID":"ec189118-e07f-4ba5-a5c0-cc1910849ca6","Type":"ContainerStarted","Data":"1c7313549db70639834ad55bed32bda3ca4b4c14c5e32635e757d8ca48dc0f43"} Jul 10 18:30:20.128614 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:20.128591 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lqrqz" event={"ID":"74645fb8-cd43-4907-b0dc-3446ed74822b","Type":"ContainerStarted","Data":"a6fb0594cc84bf08458b5d5109c9763726301a3f72126b953cbcccfc42689255"} Jul 10 18:30:20.874454 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:20.874414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:20.875019 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:20.874516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:20.875019 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:20.874560 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:30:20.875019 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:20.874655 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm podName:12d0fea4-abb3-467a-8422-7e1a81d723b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:30:22.874629649 +0000 UTC m=+69.552342100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nrqp5" (UID: "12d0fea4-abb3-467a-8422-7e1a81d723b3") : configmap "kube-rbac-proxy" not found Jul 10 18:30:20.875019 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:20.874691 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:30:20.875019 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:20.874756 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls podName:12d0fea4-abb3-467a-8422-7e1a81d723b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:30:22.87473972 +0000 UTC m=+69.552452161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nrqp5" (UID: "12d0fea4-abb3-467a-8422-7e1a81d723b3") : secret "insights-runtime-extractor-tls" not found Jul 10 18:30:20.996553 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:20.996521 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gnzkb_24f9d007-8b0e-4e0f-8062-3a66675027e8/dns-node-resolver/0.log" Jul 10 18:30:21.593708 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:21.593679 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b5dpv_b7b0a074-7ac6-4e87-9667-f1d0e4920459/node-ca/0.log" Jul 10 18:30:22.401493 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:22.401461 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-75cm8_f5ff0e60-3cce-449d-ad20-f872820b48c2/serve-healthcheck-canary/0.log" Jul 10 18:30:22.894926 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:22.894896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:22.895084 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:22.894990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:22.895084 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:22.895041 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:30:22.895084 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:22.895074 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:30:22.895213 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:22.895096 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls podName:12d0fea4-abb3-467a-8422-7e1a81d723b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:30:26.895082249 +0000 UTC m=+73.572794690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nrqp5" (UID: "12d0fea4-abb3-467a-8422-7e1a81d723b3") : secret "insights-runtime-extractor-tls" not found Jul 10 18:30:22.895213 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:22.895132 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm podName:12d0fea4-abb3-467a-8422-7e1a81d723b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:30:26.895115621 +0000 UTC m=+73.572828076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nrqp5" (UID: "12d0fea4-abb3-467a-8422-7e1a81d723b3") : configmap "kube-rbac-proxy" not found Jul 10 18:30:23.142283 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:23.142238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lqrqz" event={"ID":"74645fb8-cd43-4907-b0dc-3446ed74822b","Type":"ContainerStarted","Data":"8bb22824474e87187d7ffa5c3fa014532da216e8092a2f6dc94a40ce4a1f189d"} Jul 10 18:30:23.142486 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:23.142460 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:30:23.162079 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:23.162038 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lqrqz" podStartSLOduration=66.25716239 podStartE2EDuration="1m9.162023603s" totalImagesPullingTime="2.904861213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:30:23.160074018 +0000 UTC m=+69.837786483" watchObservedRunningTime="2026-07-10 18:30:23.162023603 +0000 UTC m=+69.839736067" Jul 10 18:30:26.153257 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:26.153221 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rs2hc" event={"ID":"ec189118-e07f-4ba5-a5c0-cc1910849ca6","Type":"ContainerStarted","Data":"7f61247649e9cc73995f32208b37a12326d836a81f21c339a0cc0603b75c6f04"} Jul 10 18:30:26.171268 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:26.171225 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rs2hc" podStartSLOduration=66.660049633 podStartE2EDuration="1m12.171209758s" totalImagesPullingTime="5.511160125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:30:26.170589767 +0000 UTC m=+72.848302223" watchObservedRunningTime="2026-07-10 18:30:26.171209758 +0000 UTC m=+72.848922219" Jul 10 18:30:26.928870 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:26.928820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:26.929084 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:26.928924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:26.929084 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:26.928972 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:30:26.929084 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:26.929016 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:30:26.929084 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:26.929033 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls podName:12d0fea4-abb3-467a-8422-7e1a81d723b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:30:34.929018434 +0000 UTC m=+81.606730874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nrqp5" (UID: "12d0fea4-abb3-467a-8422-7e1a81d723b3") : secret "insights-runtime-extractor-tls" not found Jul 10 18:30:26.929084 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:26.929064 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm podName:12d0fea4-abb3-467a-8422-7e1a81d723b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:30:34.929050541 +0000 UTC m=+81.606763002 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nrqp5" (UID: "12d0fea4-abb3-467a-8422-7e1a81d723b3") : configmap "kube-rbac-proxy" not found Jul 10 18:30:34.992071 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:34.992027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:34.992686 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:34.992098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:34.992762 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:34.992715 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12d0fea4-abb3-467a-8422-7e1a81d723b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:34.994367 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:34.994338 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12d0fea4-abb3-467a-8422-7e1a81d723b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrqp5\" (UID: \"12d0fea4-abb3-467a-8422-7e1a81d723b3\") " pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:35.103549 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:35.103510 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nrqp5" Jul 10 18:30:35.222773 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:35.222746 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nrqp5"] Jul 10 18:30:35.225604 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:30:35.225582 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d0fea4_abb3_467a_8422_7e1a81d723b3.slice/crio-dc56a749743a32bd7ec7c8159456161e643eaff9d234cf54718322c4c0c293f5 WatchSource:0}: Error finding container dc56a749743a32bd7ec7c8159456161e643eaff9d234cf54718322c4c0c293f5: Status 404 returned error can't find the container with id dc56a749743a32bd7ec7c8159456161e643eaff9d234cf54718322c4c0c293f5 Jul 10 18:30:36.185962 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:36.185822 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrqp5" event={"ID":"12d0fea4-abb3-467a-8422-7e1a81d723b3","Type":"ContainerStarted","Data":"dc56a749743a32bd7ec7c8159456161e643eaff9d234cf54718322c4c0c293f5"} Jul 10 18:30:39.199035 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:39.198999 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrqp5" event={"ID":"12d0fea4-abb3-467a-8422-7e1a81d723b3","Type":"ContainerStarted","Data":"10853ed714867f339ac055eac21552448ef83da9c61d55593ed9d05c682af3f0"} Jul 10 18:30:40.205998 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:40.205964 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrqp5" event={"ID":"12d0fea4-abb3-467a-8422-7e1a81d723b3","Type":"ContainerStarted","Data":"76378b8e5c49c43608e6a1e54d401e703acddf43fcaa1b4bf24aa560f1f3b312"} Jul 10 18:30:42.214510 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:42.214476 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrqp5" event={"ID":"12d0fea4-abb3-467a-8422-7e1a81d723b3","Type":"ContainerStarted","Data":"7d14bfb8599955a3506cf4f28a952db291cd4935571a398e381c9607802fe5ca"} Jul 10 18:30:42.231414 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:42.231368 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nrqp5" podStartSLOduration=16.754233346 podStartE2EDuration="23.231337082s" totalImagesPullingTime="6.477103736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:30:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:30:42.23026463 +0000 UTC m=+88.907977126" watchObservedRunningTime="2026-07-10 18:30:42.231337082 +0000 UTC m=+88.909049543" Jul 10 18:30:52.128108 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:52.128061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:30:52.128575 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:52.128132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:30:52.130323 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:52.130302 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fca207e-d050-4f78-beee-b9f61906bf18-metrics-tls\") pod \"dns-default-qlqq4\" (UID: \"4fca207e-d050-4f78-beee-b9f61906bf18\") " pod="openshift-dns/dns-default-qlqq4" Jul 10 18:30:52.130544 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:52.130525 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls\") pod \"image-registry-6bdbf8d7f4-ztxbb\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:30:52.412543 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:52.412467 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6l98j\"" Jul 10 18:30:52.417340 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:52.417322 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hf98r\"" Jul 10 18:30:52.420216 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:52.420201 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:30:52.425547 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:52.425533 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qlqq4" Jul 10 18:30:52.554319 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:52.554287 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb"] Jul 10 18:30:52.557813 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:30:52.557779 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55555f8f_ae9d_48eb_897f_f1fcc223a5da.slice/crio-81be2d7b770ecf8a8e4f75357b5f04934c34fb89e801174650ba674ca664aaa8 WatchSource:0}: Error finding container 81be2d7b770ecf8a8e4f75357b5f04934c34fb89e801174650ba674ca664aaa8: Status 404 returned error can't find the container with id 81be2d7b770ecf8a8e4f75357b5f04934c34fb89e801174650ba674ca664aaa8 Jul 10 18:30:52.565874 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:52.565849 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qlqq4"] Jul 10 18:30:52.568947 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:30:52.568926 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fca207e_d050_4f78_beee_b9f61906bf18.slice/crio-e1b92f120b70f51c7266294751b9da6f288e4bc729ad23c647b2808b17cc0281 WatchSource:0}: Error finding container e1b92f120b70f51c7266294751b9da6f288e4bc729ad23c647b2808b17cc0281: Status 404 returned error can't find the container with id e1b92f120b70f51c7266294751b9da6f288e4bc729ad23c647b2808b17cc0281 Jul 10 18:30:53.249259 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:53.249222 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" event={"ID":"55555f8f-ae9d-48eb-897f-f1fcc223a5da","Type":"ContainerStarted","Data":"516d46147ae45ca2e6492b9ec082d6ccecd788c529f0b3ee52e17caf468509aa"} Jul 10 18:30:53.249746 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:53.249278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" event={"ID":"55555f8f-ae9d-48eb-897f-f1fcc223a5da","Type":"ContainerStarted","Data":"81be2d7b770ecf8a8e4f75357b5f04934c34fb89e801174650ba674ca664aaa8"} Jul 10 18:30:53.249746 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:53.249311 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:30:53.250529 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:53.250505 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qlqq4" event={"ID":"4fca207e-d050-4f78-beee-b9f61906bf18","Type":"ContainerStarted","Data":"e1b92f120b70f51c7266294751b9da6f288e4bc729ad23c647b2808b17cc0281"} Jul 10 18:30:53.271639 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:53.271587 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" podStartSLOduration=99.271574301 podStartE2EDuration="1m39.271574301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:30:53.269801917 +0000 UTC m=+99.947514380" watchObservedRunningTime="2026-07-10 18:30:53.271574301 +0000 UTC m=+99.949286760" Jul 10 18:30:54.149039 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:54.149014 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lqrqz" Jul 10 18:30:54.255766 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:54.255731 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qlqq4" event={"ID":"4fca207e-d050-4f78-beee-b9f61906bf18","Type":"ContainerStarted","Data":"ca082e0dcac8fd3b994e0bbb9ab3500eb81c7b9bd808f43dac4e669e9b2f6565"} Jul 10 18:30:54.256136 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:54.255797 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qlqq4" event={"ID":"4fca207e-d050-4f78-beee-b9f61906bf18","Type":"ContainerStarted","Data":"0d971b56315933fae9ae1c468f66f536c493ee08eb8f3df5c0354883ed401cdf"} Jul 10 18:30:54.256136 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:54.256032 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qlqq4" Jul 10 18:30:54.276084 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:54.276045 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qlqq4" podStartSLOduration=64.772708537 podStartE2EDuration="1m6.276030503s" totalImagesPullingTime="1.503321966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:30:54.275548084 +0000 UTC m=+100.953260545" watchObservedRunningTime="2026-07-10 18:30:54.276030503 +0000 UTC m=+100.953742965" Jul 10 18:30:57.095408 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.095375 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m26lv"] Jul 10 18:30:57.111524 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.111499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.114517 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.114496 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 18:30:57.114625 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.114497 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 18:30:57.114807 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.114788 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 18:30:57.115090 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.115076 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 18:30:57.115246 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.115232 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 18:30:57.115561 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.115545 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 18:30:57.115607 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.115561 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cvltk\"" Jul 10 18:30:57.167215 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.167185 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d51265c6-f3cf-4982-85a9-327c4db64d6f-root\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.167215 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.167217 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-textfile\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.167425 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.167237 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d51265c6-f3cf-4982-85a9-327c4db64d6f-sys\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.167425 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.167252 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-wtmp\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.167425 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.167336 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d51265c6-f3cf-4982-85a9-327c4db64d6f-metrics-client-ca\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.167425 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.167386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-accelerators-collector-config\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.167547 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.167453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86hl7\" (UniqueName: \"kubernetes.io/projected/d51265c6-f3cf-4982-85a9-327c4db64d6f-kube-api-access-86hl7\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.167547 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.167481 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.167547 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.167501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-tls\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.267982 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.267950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d51265c6-f3cf-4982-85a9-327c4db64d6f-metrics-client-ca\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.267982 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.267984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-accelerators-collector-config\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.268175 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.268023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86hl7\" (UniqueName: \"kubernetes.io/projected/d51265c6-f3cf-4982-85a9-327c4db64d6f-kube-api-access-86hl7\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.268240 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.268223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.268275 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.268256 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-tls\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.268312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.268292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d51265c6-f3cf-4982-85a9-327c4db64d6f-root\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.268312 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.268310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-textfile\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.268462 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.268333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d51265c6-f3cf-4982-85a9-327c4db64d6f-sys\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.268462 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.268377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-wtmp\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.268462 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.268416 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d51265c6-f3cf-4982-85a9-327c4db64d6f-sys\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.268462 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:57.268429 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 18:30:57.268646 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.268474 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d51265c6-f3cf-4982-85a9-327c4db64d6f-root\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.268646 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:30:57.268492 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-tls podName:d51265c6-f3cf-4982-85a9-327c4db64d6f nodeName:}" failed. No retries permitted until 2026-07-10 18:30:57.768472937 +0000 UTC m=+104.446185377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-tls") pod "node-exporter-m26lv" (UID: "d51265c6-f3cf-4982-85a9-327c4db64d6f") : secret "node-exporter-tls" not found Jul 10 18:30:57.268646 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.268547 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-wtmp\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.268646 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.268612 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-textfile\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.268816 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.268664 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d51265c6-f3cf-4982-85a9-327c4db64d6f-metrics-client-ca\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.268816 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.268685 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-accelerators-collector-config\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.270581 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.270561 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.276421 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.276402 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86hl7\" (UniqueName: \"kubernetes.io/projected/d51265c6-f3cf-4982-85a9-327c4db64d6f-kube-api-access-86hl7\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.772154 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.772113 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-tls\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:57.774478 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:57.774458 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d51265c6-f3cf-4982-85a9-327c4db64d6f-node-exporter-tls\") pod \"node-exporter-m26lv\" (UID: \"d51265c6-f3cf-4982-85a9-327c4db64d6f\") " pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:58.020539 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:58.020503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m26lv" Jul 10 18:30:58.030338 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:30:58.030311 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51265c6_f3cf_4982_85a9_327c4db64d6f.slice/crio-86c4017670c42ba6be83143086e9a69d9ebda373a9f534f143ccca9153f1103b WatchSource:0}: Error finding container 86c4017670c42ba6be83143086e9a69d9ebda373a9f534f143ccca9153f1103b: Status 404 returned error can't find the container with id 86c4017670c42ba6be83143086e9a69d9ebda373a9f534f143ccca9153f1103b Jul 10 18:30:58.271798 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:58.271768 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m26lv" event={"ID":"d51265c6-f3cf-4982-85a9-327c4db64d6f","Type":"ContainerStarted","Data":"86c4017670c42ba6be83143086e9a69d9ebda373a9f534f143ccca9153f1103b"} Jul 10 18:30:59.276327 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:59.276245 2576 generic.go:356] "Generic (PLEG): container finished" podID="d51265c6-f3cf-4982-85a9-327c4db64d6f" containerID="d6aadfacd56705ae742df2fadeedcab74eb003c84017d09502035a7a4d34047b" exitCode=0 Jul 10 18:30:59.276327 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:30:59.276316 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m26lv" event={"ID":"d51265c6-f3cf-4982-85a9-327c4db64d6f","Type":"ContainerDied","Data":"d6aadfacd56705ae742df2fadeedcab74eb003c84017d09502035a7a4d34047b"} Jul 10 18:31:00.282598 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:00.282563 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m26lv" event={"ID":"d51265c6-f3cf-4982-85a9-327c4db64d6f","Type":"ContainerStarted","Data":"f9456c9e7f3701bbcfd6c59f030881702aec8aee063e007ed7c5f0fcd138148e"} Jul 10 18:31:00.283072 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:00.282613 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m26lv" event={"ID":"d51265c6-f3cf-4982-85a9-327c4db64d6f","Type":"ContainerStarted","Data":"727c2f9adce8d0afcd985b90b45e209baffc5a0245b40ecddc9f314db7a856e1"} Jul 10 18:31:00.304740 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:00.304699 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m26lv" podStartSLOduration=2.446854681 podStartE2EDuration="3.304684396s" totalImagesPullingTime="795.930622ms" totalInitContainerRuntime="61.899093ms" isStatefulPod=true podCreationTimestamp="2026-07-10 18:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:31:00.303169742 +0000 UTC m=+106.980882201" watchObservedRunningTime="2026-07-10 18:31:00.304684396 +0000 UTC m=+106.982396858" Jul 10 18:31:04.263462 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:04.263433 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qlqq4" Jul 10 18:31:12.424995 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:12.424962 2576 patch_prober.go:28] interesting pod/image-registry-6bdbf8d7f4-ztxbb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 18:31:12.425373 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:12.425014 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" podUID="55555f8f-ae9d-48eb-897f-f1fcc223a5da" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 18:31:13.973118 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:13.973084 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb"] Jul 10 18:31:13.977455 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:13.977432 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:31:23.583874 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:23.583838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs\") pod \"network-metrics-daemon-xlffx\" (UID: \"43310758-62c8-4c7b-8c6b-4fe08aa348d1\") " pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:31:23.586239 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:23.586214 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43310758-62c8-4c7b-8c6b-4fe08aa348d1-metrics-certs\") pod \"network-metrics-daemon-xlffx\" (UID: \"43310758-62c8-4c7b-8c6b-4fe08aa348d1\") " pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:31:23.666203 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:23.666179 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hhh94\"" Jul 10 18:31:23.674275 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:23.674251 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xlffx" Jul 10 18:31:23.819977 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:23.819925 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xlffx"] Jul 10 18:31:23.822500 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:31:23.822454 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43310758_62c8_4c7b_8c6b_4fe08aa348d1.slice/crio-d21782ce15131171c03c19b467e3118f8c1b6b81e04db90e94133e925d499ec5 WatchSource:0}: Error finding container d21782ce15131171c03c19b467e3118f8c1b6b81e04db90e94133e925d499ec5: Status 404 returned error can't find the container with id d21782ce15131171c03c19b467e3118f8c1b6b81e04db90e94133e925d499ec5 Jul 10 18:31:24.361630 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:24.361592 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xlffx" event={"ID":"43310758-62c8-4c7b-8c6b-4fe08aa348d1","Type":"ContainerStarted","Data":"d21782ce15131171c03c19b467e3118f8c1b6b81e04db90e94133e925d499ec5"} Jul 10 18:31:25.366931 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:25.366892 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xlffx" event={"ID":"43310758-62c8-4c7b-8c6b-4fe08aa348d1","Type":"ContainerStarted","Data":"e9c043331d85bdcca92a08e3a9ec2d55cec2f5c743865573a3766d0003f6cff2"} Jul 10 18:31:25.367330 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:25.366943 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xlffx" event={"ID":"43310758-62c8-4c7b-8c6b-4fe08aa348d1","Type":"ContainerStarted","Data":"8115e96da89974a3b82b3d503074aaa91eb7695636ffb22bfc47464dbcfbfb19"} Jul 10 18:31:25.385281 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:25.385235 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xlffx" podStartSLOduration=130.410080901 podStartE2EDuration="2m11.385218541s" totalImagesPullingTime="975.13764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:31:25.383813258 +0000 UTC m=+132.061525721" watchObservedRunningTime="2026-07-10 18:31:25.385218541 +0000 UTC m=+132.062931002" Jul 10 18:31:28.498069 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:28.498033 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" podUID="6b89a502-aa73-4b53-a31d-461bcde246f5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 18:31:38.498371 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:38.498312 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" podUID="6b89a502-aa73-4b53-a31d-461bcde246f5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 18:31:38.992472 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:38.992428 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" podUID="55555f8f-ae9d-48eb-897f-f1fcc223a5da" containerName="registry" containerID="cri-o://516d46147ae45ca2e6492b9ec082d6ccecd788c529f0b3ee52e17caf468509aa" gracePeriod=30 Jul 10 18:31:39.236236 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.236215 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:31:39.389859 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.389811 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls\") pod \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " Jul 10 18:31:39.390053 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.389875 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55555f8f-ae9d-48eb-897f-f1fcc223a5da-trusted-ca\") pod \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " Jul 10 18:31:39.390053 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.389905 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55555f8f-ae9d-48eb-897f-f1fcc223a5da-image-registry-private-configuration\") pod \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " Jul 10 18:31:39.390053 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.389924 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55555f8f-ae9d-48eb-897f-f1fcc223a5da-ca-trust-extracted\") pod \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " Jul 10 18:31:39.390053 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.389945 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55555f8f-ae9d-48eb-897f-f1fcc223a5da-installation-pull-secrets\") pod \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " Jul 10 18:31:39.390053 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.389970 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-certificates\") pod \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " Jul 10 18:31:39.390053 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.389991 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxn9m\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-kube-api-access-hxn9m\") pod \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " Jul 10 18:31:39.390053 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.390016 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-bound-sa-token\") pod \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\" (UID: \"55555f8f-ae9d-48eb-897f-f1fcc223a5da\") " Jul 10 18:31:39.391012 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.390561 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "55555f8f-ae9d-48eb-897f-f1fcc223a5da" (UID: "55555f8f-ae9d-48eb-897f-f1fcc223a5da"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:31:39.391012 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.390682 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55555f8f-ae9d-48eb-897f-f1fcc223a5da-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "55555f8f-ae9d-48eb-897f-f1fcc223a5da" (UID: "55555f8f-ae9d-48eb-897f-f1fcc223a5da"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:31:39.392574 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.392521 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55555f8f-ae9d-48eb-897f-f1fcc223a5da-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "55555f8f-ae9d-48eb-897f-f1fcc223a5da" (UID: "55555f8f-ae9d-48eb-897f-f1fcc223a5da"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:31:39.392707 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.392581 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "55555f8f-ae9d-48eb-897f-f1fcc223a5da" (UID: "55555f8f-ae9d-48eb-897f-f1fcc223a5da"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:31:39.392707 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.392643 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55555f8f-ae9d-48eb-897f-f1fcc223a5da-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "55555f8f-ae9d-48eb-897f-f1fcc223a5da" (UID: "55555f8f-ae9d-48eb-897f-f1fcc223a5da"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:31:39.392839 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.392705 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-kube-api-access-hxn9m" (OuterVolumeSpecName: "kube-api-access-hxn9m") pod "55555f8f-ae9d-48eb-897f-f1fcc223a5da" (UID: "55555f8f-ae9d-48eb-897f-f1fcc223a5da"). InnerVolumeSpecName "kube-api-access-hxn9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:31:39.392839 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.392777 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "55555f8f-ae9d-48eb-897f-f1fcc223a5da" (UID: "55555f8f-ae9d-48eb-897f-f1fcc223a5da"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:31:39.398781 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.398758 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55555f8f-ae9d-48eb-897f-f1fcc223a5da-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "55555f8f-ae9d-48eb-897f-f1fcc223a5da" (UID: "55555f8f-ae9d-48eb-897f-f1fcc223a5da"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:31:39.415128 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.415101 2576 generic.go:356] "Generic (PLEG): container finished" podID="55555f8f-ae9d-48eb-897f-f1fcc223a5da" containerID="516d46147ae45ca2e6492b9ec082d6ccecd788c529f0b3ee52e17caf468509aa" exitCode=0 Jul 10 18:31:39.415251 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.415144 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" Jul 10 18:31:39.415251 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.415156 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" event={"ID":"55555f8f-ae9d-48eb-897f-f1fcc223a5da","Type":"ContainerDied","Data":"516d46147ae45ca2e6492b9ec082d6ccecd788c529f0b3ee52e17caf468509aa"} Jul 10 18:31:39.415251 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.415211 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb" event={"ID":"55555f8f-ae9d-48eb-897f-f1fcc223a5da","Type":"ContainerDied","Data":"81be2d7b770ecf8a8e4f75357b5f04934c34fb89e801174650ba674ca664aaa8"} Jul 10 18:31:39.415251 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.415240 2576 scope.go:117] "RemoveContainer" containerID="516d46147ae45ca2e6492b9ec082d6ccecd788c529f0b3ee52e17caf468509aa" Jul 10 18:31:39.424647 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.424629 2576 scope.go:117] "RemoveContainer" containerID="516d46147ae45ca2e6492b9ec082d6ccecd788c529f0b3ee52e17caf468509aa" Jul 10 18:31:39.424923 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:31:39.424904 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516d46147ae45ca2e6492b9ec082d6ccecd788c529f0b3ee52e17caf468509aa\": container with ID starting with 516d46147ae45ca2e6492b9ec082d6ccecd788c529f0b3ee52e17caf468509aa not found: ID does not exist" containerID="516d46147ae45ca2e6492b9ec082d6ccecd788c529f0b3ee52e17caf468509aa" Jul 10 18:31:39.424968 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.424932 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516d46147ae45ca2e6492b9ec082d6ccecd788c529f0b3ee52e17caf468509aa"} err="failed to get container status \"516d46147ae45ca2e6492b9ec082d6ccecd788c529f0b3ee52e17caf468509aa\": rpc error: code = NotFound desc = could not find container \"516d46147ae45ca2e6492b9ec082d6ccecd788c529f0b3ee52e17caf468509aa\": container with ID starting with 516d46147ae45ca2e6492b9ec082d6ccecd788c529f0b3ee52e17caf468509aa not found: ID does not exist" Jul 10 18:31:39.437914 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.437893 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb"] Jul 10 18:31:39.443873 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.443854 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6bdbf8d7f4-ztxbb"] Jul 10 18:31:39.491216 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.491190 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55555f8f-ae9d-48eb-897f-f1fcc223a5da-trusted-ca\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:31:39.491216 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.491211 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55555f8f-ae9d-48eb-897f-f1fcc223a5da-image-registry-private-configuration\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:31:39.491379 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.491221 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55555f8f-ae9d-48eb-897f-f1fcc223a5da-ca-trust-extracted\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:31:39.491379 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.491232 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55555f8f-ae9d-48eb-897f-f1fcc223a5da-installation-pull-secrets\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:31:39.491379 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.491240 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-certificates\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:31:39.491379 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.491248 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxn9m\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-kube-api-access-hxn9m\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:31:39.491379 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.491257 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-bound-sa-token\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:31:39.491379 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.491265 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55555f8f-ae9d-48eb-897f-f1fcc223a5da-registry-tls\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:31:39.842844 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:39.842814 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55555f8f-ae9d-48eb-897f-f1fcc223a5da" path="/var/lib/kubelet/pods/55555f8f-ae9d-48eb-897f-f1fcc223a5da/volumes" Jul 10 18:31:43.987720 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:43.987690 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m26lv_d51265c6-f3cf-4982-85a9-327c4db64d6f/init-textfile/0.log" Jul 10 18:31:44.188458 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:44.188429 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m26lv_d51265c6-f3cf-4982-85a9-327c4db64d6f/node-exporter/0.log" Jul 10 18:31:44.387893 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:44.387867 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m26lv_d51265c6-f3cf-4982-85a9-327c4db64d6f/kube-rbac-proxy/0.log" Jul 10 18:31:48.497520 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:48.497480 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" podUID="6b89a502-aa73-4b53-a31d-461bcde246f5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 18:31:48.497970 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:48.497550 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" Jul 10 18:31:48.498033 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:48.498008 2576 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"ff1048e908d40da28925d99828ad594f76ad639fd1f656e56bbbe7ea7e0947f6"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 10 18:31:48.498076 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:48.498047 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" podUID="6b89a502-aa73-4b53-a31d-461bcde246f5" containerName="service-proxy" containerID="cri-o://ff1048e908d40da28925d99828ad594f76ad639fd1f656e56bbbe7ea7e0947f6" gracePeriod=30 Jul 10 18:31:49.453124 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:49.453091 2576 generic.go:356] "Generic (PLEG): container finished" podID="6b89a502-aa73-4b53-a31d-461bcde246f5" containerID="ff1048e908d40da28925d99828ad594f76ad639fd1f656e56bbbe7ea7e0947f6" exitCode=2 Jul 10 18:31:49.453335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:49.453165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" event={"ID":"6b89a502-aa73-4b53-a31d-461bcde246f5","Type":"ContainerDied","Data":"ff1048e908d40da28925d99828ad594f76ad639fd1f656e56bbbe7ea7e0947f6"} Jul 10 18:31:49.453335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:49.453220 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7b7d9cf-8w9df" event={"ID":"6b89a502-aa73-4b53-a31d-461bcde246f5","Type":"ContainerStarted","Data":"264dac0de8a85a0712be442759aead902b323f86caa2927f2b7ebbd9608f7c81"} Jul 10 18:31:50.988654 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:31:50.988627 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-75cm8_f5ff0e60-3cce-449d-ad20-f872820b48c2/serve-healthcheck-canary/0.log" Jul 10 18:34:13.739240 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:34:13.739207 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:34:13.744737 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:34:13.744713 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 18:34:13.746682 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:34:13.746663 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:35:53.613130 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.613097 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp"] Jul 10 18:35:53.613611 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.613309 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55555f8f-ae9d-48eb-897f-f1fcc223a5da" containerName="registry" Jul 10 18:35:53.613611 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.613318 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="55555f8f-ae9d-48eb-897f-f1fcc223a5da" containerName="registry" Jul 10 18:35:53.613611 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.613376 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="55555f8f-ae9d-48eb-897f-f1fcc223a5da" containerName="registry" Jul 10 18:35:53.616217 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.616201 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" Jul 10 18:35:53.619385 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.619366 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 18:35:53.619939 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.619925 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-tttn7\"" Jul 10 18:35:53.619999 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.619957 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 18:35:53.634255 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.634229 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp"] Jul 10 18:35:53.782374 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.782325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26a66057-d751-4143-af9a-8f378d0a4b45-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp\" (UID: \"26a66057-d751-4143-af9a-8f378d0a4b45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" Jul 10 18:35:53.782547 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.782392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26a66057-d751-4143-af9a-8f378d0a4b45-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp\" (UID: \"26a66057-d751-4143-af9a-8f378d0a4b45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" Jul 10 18:35:53.782547 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.782447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffxws\" (UniqueName: \"kubernetes.io/projected/26a66057-d751-4143-af9a-8f378d0a4b45-kube-api-access-ffxws\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp\" (UID: \"26a66057-d751-4143-af9a-8f378d0a4b45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" Jul 10 18:35:53.883394 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.883293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26a66057-d751-4143-af9a-8f378d0a4b45-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp\" (UID: \"26a66057-d751-4143-af9a-8f378d0a4b45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" Jul 10 18:35:53.883394 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.883344 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26a66057-d751-4143-af9a-8f378d0a4b45-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp\" (UID: \"26a66057-d751-4143-af9a-8f378d0a4b45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" Jul 10 18:35:53.883598 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.883418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffxws\" (UniqueName: \"kubernetes.io/projected/26a66057-d751-4143-af9a-8f378d0a4b45-kube-api-access-ffxws\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp\" (UID: \"26a66057-d751-4143-af9a-8f378d0a4b45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" Jul 10 18:35:53.883807 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.883783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26a66057-d751-4143-af9a-8f378d0a4b45-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp\" (UID: \"26a66057-d751-4143-af9a-8f378d0a4b45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" Jul 10 18:35:53.883860 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.883786 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26a66057-d751-4143-af9a-8f378d0a4b45-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp\" (UID: \"26a66057-d751-4143-af9a-8f378d0a4b45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" Jul 10 18:35:53.892041 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.892014 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffxws\" (UniqueName: \"kubernetes.io/projected/26a66057-d751-4143-af9a-8f378d0a4b45-kube-api-access-ffxws\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp\" (UID: \"26a66057-d751-4143-af9a-8f378d0a4b45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" Jul 10 18:35:53.924210 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:53.924180 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" Jul 10 18:35:54.044144 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:54.044120 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp"] Jul 10 18:35:54.046490 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:35:54.046461 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a66057_d751_4143_af9a_8f378d0a4b45.slice/crio-9d6d3be900a2aa4af8af2d7991c8a670a0fd964ed45c9acfea76431a7e4d70ff WatchSource:0}: Error finding container 9d6d3be900a2aa4af8af2d7991c8a670a0fd964ed45c9acfea76431a7e4d70ff: Status 404 returned error can't find the container with id 9d6d3be900a2aa4af8af2d7991c8a670a0fd964ed45c9acfea76431a7e4d70ff Jul 10 18:35:54.048499 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:54.048482 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 18:35:54.218822 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:35:54.218744 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" event={"ID":"26a66057-d751-4143-af9a-8f378d0a4b45","Type":"ContainerStarted","Data":"9d6d3be900a2aa4af8af2d7991c8a670a0fd964ed45c9acfea76431a7e4d70ff"} Jul 10 18:36:01.244000 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:01.243965 2576 generic.go:356] "Generic (PLEG): container finished" podID="26a66057-d751-4143-af9a-8f378d0a4b45" containerID="9245373606aa846b7e2c9b24c4c71b850eb2578fd8c19afdb17b50b0f141d03f" exitCode=0 Jul 10 18:36:01.244389 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:01.244053 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" event={"ID":"26a66057-d751-4143-af9a-8f378d0a4b45","Type":"ContainerDied","Data":"9245373606aa846b7e2c9b24c4c71b850eb2578fd8c19afdb17b50b0f141d03f"} Jul 10 18:36:04.255620 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:04.255589 2576 generic.go:356] "Generic (PLEG): container finished" podID="26a66057-d751-4143-af9a-8f378d0a4b45" containerID="2f4f21883dd1936f2e502990255b10361065f834ff8df139bc685cc7bb054233" exitCode=0 Jul 10 18:36:04.255962 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:04.255637 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" event={"ID":"26a66057-d751-4143-af9a-8f378d0a4b45","Type":"ContainerDied","Data":"2f4f21883dd1936f2e502990255b10361065f834ff8df139bc685cc7bb054233"} Jul 10 18:36:12.286005 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:12.285971 2576 generic.go:356] "Generic (PLEG): container finished" podID="26a66057-d751-4143-af9a-8f378d0a4b45" containerID="1e3a42170f55b1c580f438320903b32390b9b472a839c3c382e0d1158e67e647" exitCode=0 Jul 10 18:36:12.286392 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:12.286049 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" event={"ID":"26a66057-d751-4143-af9a-8f378d0a4b45","Type":"ContainerDied","Data":"1e3a42170f55b1c580f438320903b32390b9b472a839c3c382e0d1158e67e647"} Jul 10 18:36:13.403945 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:13.403921 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" Jul 10 18:36:13.541288 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:13.541206 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26a66057-d751-4143-af9a-8f378d0a4b45-bundle\") pod \"26a66057-d751-4143-af9a-8f378d0a4b45\" (UID: \"26a66057-d751-4143-af9a-8f378d0a4b45\") " Jul 10 18:36:13.541288 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:13.541250 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26a66057-d751-4143-af9a-8f378d0a4b45-util\") pod \"26a66057-d751-4143-af9a-8f378d0a4b45\" (UID: \"26a66057-d751-4143-af9a-8f378d0a4b45\") " Jul 10 18:36:13.541490 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:13.541315 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffxws\" (UniqueName: \"kubernetes.io/projected/26a66057-d751-4143-af9a-8f378d0a4b45-kube-api-access-ffxws\") pod \"26a66057-d751-4143-af9a-8f378d0a4b45\" (UID: \"26a66057-d751-4143-af9a-8f378d0a4b45\") " Jul 10 18:36:13.541910 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:13.541874 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a66057-d751-4143-af9a-8f378d0a4b45-bundle" (OuterVolumeSpecName: "bundle") pod "26a66057-d751-4143-af9a-8f378d0a4b45" (UID: "26a66057-d751-4143-af9a-8f378d0a4b45"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:36:13.543519 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:13.543494 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a66057-d751-4143-af9a-8f378d0a4b45-kube-api-access-ffxws" (OuterVolumeSpecName: "kube-api-access-ffxws") pod "26a66057-d751-4143-af9a-8f378d0a4b45" (UID: "26a66057-d751-4143-af9a-8f378d0a4b45"). InnerVolumeSpecName "kube-api-access-ffxws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:36:13.545314 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:13.545295 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a66057-d751-4143-af9a-8f378d0a4b45-util" (OuterVolumeSpecName: "util") pod "26a66057-d751-4143-af9a-8f378d0a4b45" (UID: "26a66057-d751-4143-af9a-8f378d0a4b45"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:36:13.641986 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:13.641957 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffxws\" (UniqueName: \"kubernetes.io/projected/26a66057-d751-4143-af9a-8f378d0a4b45-kube-api-access-ffxws\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:36:13.641986 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:13.641982 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26a66057-d751-4143-af9a-8f378d0a4b45-bundle\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:36:13.641986 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:13.641990 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26a66057-d751-4143-af9a-8f378d0a4b45-util\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:36:14.294721 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:14.294642 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" event={"ID":"26a66057-d751-4143-af9a-8f378d0a4b45","Type":"ContainerDied","Data":"9d6d3be900a2aa4af8af2d7991c8a670a0fd964ed45c9acfea76431a7e4d70ff"} Jul 10 18:36:14.294721 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:14.294661 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409zhxp" Jul 10 18:36:14.294721 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:14.294686 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d6d3be900a2aa4af8af2d7991c8a670a0fd964ed45c9acfea76431a7e4d70ff" Jul 10 18:36:20.207824 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.207787 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wddkw"] Jul 10 18:36:20.208187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.208002 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a66057-d751-4143-af9a-8f378d0a4b45" containerName="extract" Jul 10 18:36:20.208187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.208014 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a66057-d751-4143-af9a-8f378d0a4b45" containerName="extract" Jul 10 18:36:20.208187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.208024 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a66057-d751-4143-af9a-8f378d0a4b45" containerName="util" Jul 10 18:36:20.208187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.208030 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a66057-d751-4143-af9a-8f378d0a4b45" containerName="util" Jul 10 18:36:20.208187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.208036 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a66057-d751-4143-af9a-8f378d0a4b45" containerName="pull" Jul 10 18:36:20.208187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.208041 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a66057-d751-4143-af9a-8f378d0a4b45" containerName="pull" Jul 10 18:36:20.208187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.208079 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a66057-d751-4143-af9a-8f378d0a4b45" containerName="extract" Jul 10 18:36:20.210979 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.210963 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wddkw" Jul 10 18:36:20.214263 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.214246 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 10 18:36:20.214627 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.214604 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:36:20.214721 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.214639 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-m8hfm\"" Jul 10 18:36:20.225907 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.225886 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wddkw"] Jul 10 18:36:20.286488 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.286455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a08b8159-4c2d-445b-bee2-7107761073f2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wddkw\" (UID: \"a08b8159-4c2d-445b-bee2-7107761073f2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wddkw" Jul 10 18:36:20.286634 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.286504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5w6g\" (UniqueName: \"kubernetes.io/projected/a08b8159-4c2d-445b-bee2-7107761073f2-kube-api-access-g5w6g\") pod \"cert-manager-operator-controller-manager-55c444bf86-wddkw\" (UID: \"a08b8159-4c2d-445b-bee2-7107761073f2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wddkw" Jul 10 18:36:20.387683 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.387643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5w6g\" (UniqueName: \"kubernetes.io/projected/a08b8159-4c2d-445b-bee2-7107761073f2-kube-api-access-g5w6g\") pod \"cert-manager-operator-controller-manager-55c444bf86-wddkw\" (UID: \"a08b8159-4c2d-445b-bee2-7107761073f2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wddkw" Jul 10 18:36:20.387893 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.387716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a08b8159-4c2d-445b-bee2-7107761073f2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wddkw\" (UID: \"a08b8159-4c2d-445b-bee2-7107761073f2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wddkw" Jul 10 18:36:20.388052 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.388021 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a08b8159-4c2d-445b-bee2-7107761073f2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wddkw\" (UID: \"a08b8159-4c2d-445b-bee2-7107761073f2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wddkw" Jul 10 18:36:20.396476 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.396446 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5w6g\" (UniqueName: \"kubernetes.io/projected/a08b8159-4c2d-445b-bee2-7107761073f2-kube-api-access-g5w6g\") pod \"cert-manager-operator-controller-manager-55c444bf86-wddkw\" (UID: \"a08b8159-4c2d-445b-bee2-7107761073f2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wddkw" Jul 10 18:36:20.519496 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.519413 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wddkw" Jul 10 18:36:20.645670 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:20.645653 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wddkw"] Jul 10 18:36:20.648033 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:36:20.648003 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08b8159_4c2d_445b_bee2_7107761073f2.slice/crio-b04ce5d45535c3b7f79cfb9860fad97adff3f0356e0e62badb930faa017ff874 WatchSource:0}: Error finding container b04ce5d45535c3b7f79cfb9860fad97adff3f0356e0e62badb930faa017ff874: Status 404 returned error can't find the container with id b04ce5d45535c3b7f79cfb9860fad97adff3f0356e0e62badb930faa017ff874 Jul 10 18:36:21.316656 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:21.316623 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wddkw" event={"ID":"a08b8159-4c2d-445b-bee2-7107761073f2","Type":"ContainerStarted","Data":"b04ce5d45535c3b7f79cfb9860fad97adff3f0356e0e62badb930faa017ff874"} Jul 10 18:36:23.324403 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:23.324345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wddkw" event={"ID":"a08b8159-4c2d-445b-bee2-7107761073f2","Type":"ContainerStarted","Data":"866a7ac8b55b466e59b722462d44df93a3f552111cd6252f73927217535134c2"} Jul 10 18:36:23.347472 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:23.347428 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wddkw" podStartSLOduration=1.578671426 podStartE2EDuration="3.347395235s" totalImagesPullingTime="1.768723809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:36:23.34562752 +0000 UTC m=+430.023339983" watchObservedRunningTime="2026-07-10 18:36:23.347395235 +0000 UTC m=+430.025107697" Jul 10 18:36:24.332505 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.332473 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5"] Jul 10 18:36:24.334715 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.334699 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" Jul 10 18:36:24.337687 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.337669 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 18:36:24.337808 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.337748 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-tttn7\"" Jul 10 18:36:24.337808 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.337775 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 18:36:24.344607 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.344587 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5"] Jul 10 18:36:24.414099 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.414068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2abbf423-fa6d-4aef-85a4-f2288a7738e9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5\" (UID: \"2abbf423-fa6d-4aef-85a4-f2288a7738e9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" Jul 10 18:36:24.414317 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.414129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2abbf423-fa6d-4aef-85a4-f2288a7738e9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5\" (UID: \"2abbf423-fa6d-4aef-85a4-f2288a7738e9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" Jul 10 18:36:24.414317 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.414159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9k86\" (UniqueName: \"kubernetes.io/projected/2abbf423-fa6d-4aef-85a4-f2288a7738e9-kube-api-access-j9k86\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5\" (UID: \"2abbf423-fa6d-4aef-85a4-f2288a7738e9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" Jul 10 18:36:24.514468 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.514436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2abbf423-fa6d-4aef-85a4-f2288a7738e9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5\" (UID: \"2abbf423-fa6d-4aef-85a4-f2288a7738e9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" Jul 10 18:36:24.514635 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.514475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2abbf423-fa6d-4aef-85a4-f2288a7738e9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5\" (UID: \"2abbf423-fa6d-4aef-85a4-f2288a7738e9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" Jul 10 18:36:24.514635 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.514508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9k86\" (UniqueName: \"kubernetes.io/projected/2abbf423-fa6d-4aef-85a4-f2288a7738e9-kube-api-access-j9k86\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5\" (UID: \"2abbf423-fa6d-4aef-85a4-f2288a7738e9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" Jul 10 18:36:24.514827 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.514808 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2abbf423-fa6d-4aef-85a4-f2288a7738e9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5\" (UID: \"2abbf423-fa6d-4aef-85a4-f2288a7738e9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" Jul 10 18:36:24.514883 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.514836 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2abbf423-fa6d-4aef-85a4-f2288a7738e9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5\" (UID: \"2abbf423-fa6d-4aef-85a4-f2288a7738e9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" Jul 10 18:36:24.529410 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.529387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9k86\" (UniqueName: \"kubernetes.io/projected/2abbf423-fa6d-4aef-85a4-f2288a7738e9-kube-api-access-j9k86\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5\" (UID: \"2abbf423-fa6d-4aef-85a4-f2288a7738e9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" Jul 10 18:36:24.643803 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.643735 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" Jul 10 18:36:24.766083 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:24.766053 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5"] Jul 10 18:36:24.768458 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:36:24.768428 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abbf423_fa6d_4aef_85a4_f2288a7738e9.slice/crio-e9b36db733dad837b64ad5cb4feeab22a759db51d9c107a3781cbd247d30ef60 WatchSource:0}: Error finding container e9b36db733dad837b64ad5cb4feeab22a759db51d9c107a3781cbd247d30ef60: Status 404 returned error can't find the container with id e9b36db733dad837b64ad5cb4feeab22a759db51d9c107a3781cbd247d30ef60 Jul 10 18:36:25.332036 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:25.332003 2576 generic.go:356] "Generic (PLEG): container finished" podID="2abbf423-fa6d-4aef-85a4-f2288a7738e9" containerID="ca8164e7c663d8f5711c9ae035d7ee5c3f0ffa522a82b7eaccd78f6cb3c67e82" exitCode=0 Jul 10 18:36:25.332205 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:25.332086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" event={"ID":"2abbf423-fa6d-4aef-85a4-f2288a7738e9","Type":"ContainerDied","Data":"ca8164e7c663d8f5711c9ae035d7ee5c3f0ffa522a82b7eaccd78f6cb3c67e82"} Jul 10 18:36:25.332205 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:25.332137 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" event={"ID":"2abbf423-fa6d-4aef-85a4-f2288a7738e9","Type":"ContainerStarted","Data":"e9b36db733dad837b64ad5cb4feeab22a759db51d9c107a3781cbd247d30ef60"} Jul 10 18:36:28.343875 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:28.343842 2576 generic.go:356] "Generic (PLEG): container finished" podID="2abbf423-fa6d-4aef-85a4-f2288a7738e9" containerID="a342109f3a7946b5cf9054a210953dec69d3573bcd97cd2e078b953ec6741fdb" exitCode=0 Jul 10 18:36:28.344321 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:28.343882 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" event={"ID":"2abbf423-fa6d-4aef-85a4-f2288a7738e9","Type":"ContainerDied","Data":"a342109f3a7946b5cf9054a210953dec69d3573bcd97cd2e078b953ec6741fdb"} Jul 10 18:36:29.348974 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:29.348936 2576 generic.go:356] "Generic (PLEG): container finished" podID="2abbf423-fa6d-4aef-85a4-f2288a7738e9" containerID="e24050a0da4732d5b28b70cf9ac97bfe111d1256e235ff468a05dc5356b65031" exitCode=0 Jul 10 18:36:29.349338 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:29.349007 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" event={"ID":"2abbf423-fa6d-4aef-85a4-f2288a7738e9","Type":"ContainerDied","Data":"e24050a0da4732d5b28b70cf9ac97bfe111d1256e235ff468a05dc5356b65031"} Jul 10 18:36:30.477135 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:30.477110 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" Jul 10 18:36:30.560796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:30.560766 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2abbf423-fa6d-4aef-85a4-f2288a7738e9-bundle\") pod \"2abbf423-fa6d-4aef-85a4-f2288a7738e9\" (UID: \"2abbf423-fa6d-4aef-85a4-f2288a7738e9\") " Jul 10 18:36:30.560796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:30.560797 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9k86\" (UniqueName: \"kubernetes.io/projected/2abbf423-fa6d-4aef-85a4-f2288a7738e9-kube-api-access-j9k86\") pod \"2abbf423-fa6d-4aef-85a4-f2288a7738e9\" (UID: \"2abbf423-fa6d-4aef-85a4-f2288a7738e9\") " Jul 10 18:36:30.560986 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:30.560828 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2abbf423-fa6d-4aef-85a4-f2288a7738e9-util\") pod \"2abbf423-fa6d-4aef-85a4-f2288a7738e9\" (UID: \"2abbf423-fa6d-4aef-85a4-f2288a7738e9\") " Jul 10 18:36:30.561156 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:30.561130 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abbf423-fa6d-4aef-85a4-f2288a7738e9-bundle" (OuterVolumeSpecName: "bundle") pod "2abbf423-fa6d-4aef-85a4-f2288a7738e9" (UID: "2abbf423-fa6d-4aef-85a4-f2288a7738e9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:36:30.562814 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:30.562788 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abbf423-fa6d-4aef-85a4-f2288a7738e9-kube-api-access-j9k86" (OuterVolumeSpecName: "kube-api-access-j9k86") pod "2abbf423-fa6d-4aef-85a4-f2288a7738e9" (UID: "2abbf423-fa6d-4aef-85a4-f2288a7738e9"). InnerVolumeSpecName "kube-api-access-j9k86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:36:30.565375 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:30.565337 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abbf423-fa6d-4aef-85a4-f2288a7738e9-util" (OuterVolumeSpecName: "util") pod "2abbf423-fa6d-4aef-85a4-f2288a7738e9" (UID: "2abbf423-fa6d-4aef-85a4-f2288a7738e9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:36:30.661289 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:30.661216 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2abbf423-fa6d-4aef-85a4-f2288a7738e9-bundle\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:36:30.661289 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:30.661242 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9k86\" (UniqueName: \"kubernetes.io/projected/2abbf423-fa6d-4aef-85a4-f2288a7738e9-kube-api-access-j9k86\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:36:30.661289 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:30.661251 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2abbf423-fa6d-4aef-85a4-f2288a7738e9-util\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:36:31.356844 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:31.356820 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" Jul 10 18:36:31.357004 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:31.356818 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7clz5" event={"ID":"2abbf423-fa6d-4aef-85a4-f2288a7738e9","Type":"ContainerDied","Data":"e9b36db733dad837b64ad5cb4feeab22a759db51d9c107a3781cbd247d30ef60"} Jul 10 18:36:31.357004 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:31.356945 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9b36db733dad837b64ad5cb4feeab22a759db51d9c107a3781cbd247d30ef60" Jul 10 18:36:35.210416 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.210383 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-59gbd"] Jul 10 18:36:35.210760 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.210613 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abbf423-fa6d-4aef-85a4-f2288a7738e9" containerName="pull" Jul 10 18:36:35.210760 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.210625 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abbf423-fa6d-4aef-85a4-f2288a7738e9" containerName="pull" Jul 10 18:36:35.210760 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.210640 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abbf423-fa6d-4aef-85a4-f2288a7738e9" containerName="util" Jul 10 18:36:35.210760 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.210646 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abbf423-fa6d-4aef-85a4-f2288a7738e9" containerName="util" Jul 10 18:36:35.210760 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.210653 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abbf423-fa6d-4aef-85a4-f2288a7738e9" containerName="extract" Jul 10 18:36:35.210760 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.210659 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abbf423-fa6d-4aef-85a4-f2288a7738e9" containerName="extract" Jul 10 18:36:35.210760 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.210701 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2abbf423-fa6d-4aef-85a4-f2288a7738e9" containerName="extract" Jul 10 18:36:35.258013 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.257987 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-59gbd"] Jul 10 18:36:35.258158 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.258074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-59gbd" Jul 10 18:36:35.260661 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.260631 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 18:36:35.260796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.260720 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 18:36:35.260862 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.260812 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-r5ntv\"" Jul 10 18:36:35.291149 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.291124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d83350a-b4d2-4cf6-9a6d-9e7a75a148e0-bound-sa-token\") pod \"cert-manager-55b4d954d5-59gbd\" (UID: \"3d83350a-b4d2-4cf6-9a6d-9e7a75a148e0\") " pod="cert-manager/cert-manager-55b4d954d5-59gbd" Jul 10 18:36:35.291252 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.291152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8f6v\" (UniqueName: \"kubernetes.io/projected/3d83350a-b4d2-4cf6-9a6d-9e7a75a148e0-kube-api-access-c8f6v\") pod \"cert-manager-55b4d954d5-59gbd\" (UID: \"3d83350a-b4d2-4cf6-9a6d-9e7a75a148e0\") " pod="cert-manager/cert-manager-55b4d954d5-59gbd" Jul 10 18:36:35.392465 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.392434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d83350a-b4d2-4cf6-9a6d-9e7a75a148e0-bound-sa-token\") pod \"cert-manager-55b4d954d5-59gbd\" (UID: \"3d83350a-b4d2-4cf6-9a6d-9e7a75a148e0\") " pod="cert-manager/cert-manager-55b4d954d5-59gbd" Jul 10 18:36:35.392465 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.392468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8f6v\" (UniqueName: \"kubernetes.io/projected/3d83350a-b4d2-4cf6-9a6d-9e7a75a148e0-kube-api-access-c8f6v\") pod \"cert-manager-55b4d954d5-59gbd\" (UID: \"3d83350a-b4d2-4cf6-9a6d-9e7a75a148e0\") " pod="cert-manager/cert-manager-55b4d954d5-59gbd" Jul 10 18:36:35.401607 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.401565 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d83350a-b4d2-4cf6-9a6d-9e7a75a148e0-bound-sa-token\") pod \"cert-manager-55b4d954d5-59gbd\" (UID: \"3d83350a-b4d2-4cf6-9a6d-9e7a75a148e0\") " pod="cert-manager/cert-manager-55b4d954d5-59gbd" Jul 10 18:36:35.401730 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.401655 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8f6v\" (UniqueName: \"kubernetes.io/projected/3d83350a-b4d2-4cf6-9a6d-9e7a75a148e0-kube-api-access-c8f6v\") pod \"cert-manager-55b4d954d5-59gbd\" (UID: \"3d83350a-b4d2-4cf6-9a6d-9e7a75a148e0\") " pod="cert-manager/cert-manager-55b4d954d5-59gbd" Jul 10 18:36:35.566547 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.566521 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-59gbd" Jul 10 18:36:35.739926 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:35.739899 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-59gbd"] Jul 10 18:36:35.742574 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:36:35.742549 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d83350a_b4d2_4cf6_9a6d_9e7a75a148e0.slice/crio-24443c340297d6454a180b2c5b6081a1063f51f805d8acc112323aa20a1bb3d0 WatchSource:0}: Error finding container 24443c340297d6454a180b2c5b6081a1063f51f805d8acc112323aa20a1bb3d0: Status 404 returned error can't find the container with id 24443c340297d6454a180b2c5b6081a1063f51f805d8acc112323aa20a1bb3d0 Jul 10 18:36:36.374894 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:36.374808 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-59gbd" event={"ID":"3d83350a-b4d2-4cf6-9a6d-9e7a75a148e0","Type":"ContainerStarted","Data":"24443c340297d6454a180b2c5b6081a1063f51f805d8acc112323aa20a1bb3d0"} Jul 10 18:36:38.148698 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:38.148662 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dsm6k"] Jul 10 18:36:38.150925 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:38.150905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dsm6k" Jul 10 18:36:38.154583 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:38.154559 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-d2jc2\"" Jul 10 18:36:38.154583 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:38.154564 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:36:38.155489 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:38.155473 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 18:36:38.161152 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:38.161131 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dsm6k"] Jul 10 18:36:38.214231 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:38.214195 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbgwh\" (UniqueName: \"kubernetes.io/projected/ae1ef643-3b34-449c-a8d5-4603d06458f3-kube-api-access-vbgwh\") pod \"openshift-lws-operator-fd8ccff4c-dsm6k\" (UID: \"ae1ef643-3b34-449c-a8d5-4603d06458f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dsm6k" Jul 10 18:36:38.214426 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:38.214259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae1ef643-3b34-449c-a8d5-4603d06458f3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dsm6k\" (UID: \"ae1ef643-3b34-449c-a8d5-4603d06458f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dsm6k" Jul 10 18:36:38.315534 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:38.315497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae1ef643-3b34-449c-a8d5-4603d06458f3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dsm6k\" (UID: \"ae1ef643-3b34-449c-a8d5-4603d06458f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dsm6k" Jul 10 18:36:38.315707 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:38.315559 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbgwh\" (UniqueName: \"kubernetes.io/projected/ae1ef643-3b34-449c-a8d5-4603d06458f3-kube-api-access-vbgwh\") pod \"openshift-lws-operator-fd8ccff4c-dsm6k\" (UID: \"ae1ef643-3b34-449c-a8d5-4603d06458f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dsm6k" Jul 10 18:36:38.315998 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:38.315965 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae1ef643-3b34-449c-a8d5-4603d06458f3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dsm6k\" (UID: \"ae1ef643-3b34-449c-a8d5-4603d06458f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dsm6k" Jul 10 18:36:38.325516 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:38.325488 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbgwh\" (UniqueName: \"kubernetes.io/projected/ae1ef643-3b34-449c-a8d5-4603d06458f3-kube-api-access-vbgwh\") pod \"openshift-lws-operator-fd8ccff4c-dsm6k\" (UID: \"ae1ef643-3b34-449c-a8d5-4603d06458f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dsm6k" Jul 10 18:36:38.461645 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:38.461564 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dsm6k" Jul 10 18:36:38.916022 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:38.915921 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dsm6k"] Jul 10 18:36:38.917954 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:36:38.917923 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae1ef643_3b34_449c_a8d5_4603d06458f3.slice/crio-f68c27c36cf8981155c9234e23c1702e9870d253168636b460ebc6a88bc0d247 WatchSource:0}: Error finding container f68c27c36cf8981155c9234e23c1702e9870d253168636b460ebc6a88bc0d247: Status 404 returned error can't find the container with id f68c27c36cf8981155c9234e23c1702e9870d253168636b460ebc6a88bc0d247 Jul 10 18:36:39.390450 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:39.390413 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-59gbd" event={"ID":"3d83350a-b4d2-4cf6-9a6d-9e7a75a148e0","Type":"ContainerStarted","Data":"f9f1a018b0aa86fc7fb33ae348fa8954698cce77cd82033c932ffa8ec691dd37"} Jul 10 18:36:39.391673 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:39.391647 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dsm6k" event={"ID":"ae1ef643-3b34-449c-a8d5-4603d06458f3","Type":"ContainerStarted","Data":"f68c27c36cf8981155c9234e23c1702e9870d253168636b460ebc6a88bc0d247"} Jul 10 18:36:39.410000 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:39.409954 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-59gbd" podStartSLOduration=1.332643061 podStartE2EDuration="4.409940972s" totalImagesPullingTime="3.077297911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:36:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:36:39.408706396 +0000 UTC m=+446.086418857" watchObservedRunningTime="2026-07-10 18:36:39.409940972 +0000 UTC m=+446.087653433" Jul 10 18:36:41.402968 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:41.402884 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dsm6k" event={"ID":"ae1ef643-3b34-449c-a8d5-4603d06458f3","Type":"ContainerStarted","Data":"4995a4a6f12bf84f67969d9a053ee6221e4c786c270a2949de99874767e907d6"} Jul 10 18:36:43.952656 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:43.952606 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dsm6k" podStartSLOduration=3.765088864 podStartE2EDuration="5.952586665s" totalImagesPullingTime="2.187497801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:36:41.420283877 +0000 UTC m=+448.097996341" watchObservedRunningTime="2026-07-10 18:36:43.952586665 +0000 UTC m=+450.630299128" Jul 10 18:36:43.953613 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:43.953584 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6"] Jul 10 18:36:43.971465 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:43.971439 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6"] Jul 10 18:36:43.971600 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:43.971536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" Jul 10 18:36:43.974182 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:43.974155 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-tttn7\"" Jul 10 18:36:43.974319 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:43.974158 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 18:36:43.975077 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:43.975061 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 18:36:44.058366 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:44.058322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6\" (UID: \"82ee94ff-103a-45c8-bdd6-80bb0ed4103e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" Jul 10 18:36:44.058504 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:44.058398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnskv\" (UniqueName: \"kubernetes.io/projected/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-kube-api-access-hnskv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6\" (UID: \"82ee94ff-103a-45c8-bdd6-80bb0ed4103e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" Jul 10 18:36:44.058504 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:44.058421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6\" (UID: \"82ee94ff-103a-45c8-bdd6-80bb0ed4103e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" Jul 10 18:36:44.159776 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:44.159745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6\" (UID: \"82ee94ff-103a-45c8-bdd6-80bb0ed4103e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" Jul 10 18:36:44.159958 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:44.159784 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnskv\" (UniqueName: \"kubernetes.io/projected/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-kube-api-access-hnskv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6\" (UID: \"82ee94ff-103a-45c8-bdd6-80bb0ed4103e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" Jul 10 18:36:44.159958 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:44.159802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6\" (UID: \"82ee94ff-103a-45c8-bdd6-80bb0ed4103e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" Jul 10 18:36:44.160107 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:44.160088 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6\" (UID: \"82ee94ff-103a-45c8-bdd6-80bb0ed4103e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" Jul 10 18:36:44.160182 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:44.160167 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6\" (UID: \"82ee94ff-103a-45c8-bdd6-80bb0ed4103e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" Jul 10 18:36:44.167454 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:44.167437 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnskv\" (UniqueName: \"kubernetes.io/projected/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-kube-api-access-hnskv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6\" (UID: \"82ee94ff-103a-45c8-bdd6-80bb0ed4103e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" Jul 10 18:36:44.280887 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:44.280820 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" Jul 10 18:36:44.439218 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:44.439185 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6"] Jul 10 18:36:44.442364 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:36:44.442323 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ee94ff_103a_45c8_bdd6_80bb0ed4103e.slice/crio-ea7345d3e81c98621ac74bf9f2cff4ae75772b4db43e7422f2399cbc9111247c WatchSource:0}: Error finding container ea7345d3e81c98621ac74bf9f2cff4ae75772b4db43e7422f2399cbc9111247c: Status 404 returned error can't find the container with id ea7345d3e81c98621ac74bf9f2cff4ae75772b4db43e7422f2399cbc9111247c Jul 10 18:36:45.421964 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:45.421879 2576 generic.go:356] "Generic (PLEG): container finished" podID="82ee94ff-103a-45c8-bdd6-80bb0ed4103e" containerID="27ad485d009e889608d3ca8865029aedebabe38d572d1cb9008c4c754a4654eb" exitCode=0 Jul 10 18:36:45.422328 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:45.421961 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" event={"ID":"82ee94ff-103a-45c8-bdd6-80bb0ed4103e","Type":"ContainerDied","Data":"27ad485d009e889608d3ca8865029aedebabe38d572d1cb9008c4c754a4654eb"} Jul 10 18:36:45.422328 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:45.422046 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" event={"ID":"82ee94ff-103a-45c8-bdd6-80bb0ed4103e","Type":"ContainerStarted","Data":"ea7345d3e81c98621ac74bf9f2cff4ae75772b4db43e7422f2399cbc9111247c"} Jul 10 18:36:47.431642 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:47.431610 2576 generic.go:356] "Generic (PLEG): container finished" podID="82ee94ff-103a-45c8-bdd6-80bb0ed4103e" containerID="2ef68818153852ba6b0cceb09605d8fd104affa8706d2f2bfb5d3a861d8516fa" exitCode=0 Jul 10 18:36:47.432009 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:47.431665 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" event={"ID":"82ee94ff-103a-45c8-bdd6-80bb0ed4103e","Type":"ContainerDied","Data":"2ef68818153852ba6b0cceb09605d8fd104affa8706d2f2bfb5d3a861d8516fa"} Jul 10 18:36:48.437160 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:48.437126 2576 generic.go:356] "Generic (PLEG): container finished" podID="82ee94ff-103a-45c8-bdd6-80bb0ed4103e" containerID="5d32371b6ce2c9f6231ce8214e07909531c2e83b0c752c47ffbe64d4433b3b75" exitCode=0 Jul 10 18:36:48.437563 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:48.437181 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" event={"ID":"82ee94ff-103a-45c8-bdd6-80bb0ed4103e","Type":"ContainerDied","Data":"5d32371b6ce2c9f6231ce8214e07909531c2e83b0c752c47ffbe64d4433b3b75"} Jul 10 18:36:49.567691 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:49.567665 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" Jul 10 18:36:49.597883 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:49.597860 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-bundle\") pod \"82ee94ff-103a-45c8-bdd6-80bb0ed4103e\" (UID: \"82ee94ff-103a-45c8-bdd6-80bb0ed4103e\") " Jul 10 18:36:49.598019 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:49.597945 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnskv\" (UniqueName: \"kubernetes.io/projected/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-kube-api-access-hnskv\") pod \"82ee94ff-103a-45c8-bdd6-80bb0ed4103e\" (UID: \"82ee94ff-103a-45c8-bdd6-80bb0ed4103e\") " Jul 10 18:36:49.598019 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:49.597976 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-util\") pod \"82ee94ff-103a-45c8-bdd6-80bb0ed4103e\" (UID: \"82ee94ff-103a-45c8-bdd6-80bb0ed4103e\") " Jul 10 18:36:49.598633 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:49.598612 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-bundle" (OuterVolumeSpecName: "bundle") pod "82ee94ff-103a-45c8-bdd6-80bb0ed4103e" (UID: "82ee94ff-103a-45c8-bdd6-80bb0ed4103e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:36:49.600027 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:49.599997 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-kube-api-access-hnskv" (OuterVolumeSpecName: "kube-api-access-hnskv") pod "82ee94ff-103a-45c8-bdd6-80bb0ed4103e" (UID: "82ee94ff-103a-45c8-bdd6-80bb0ed4103e"). InnerVolumeSpecName "kube-api-access-hnskv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:36:49.603915 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:49.603893 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-util" (OuterVolumeSpecName: "util") pod "82ee94ff-103a-45c8-bdd6-80bb0ed4103e" (UID: "82ee94ff-103a-45c8-bdd6-80bb0ed4103e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:36:49.699065 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:49.698995 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-bundle\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:36:49.699065 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:49.699018 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnskv\" (UniqueName: \"kubernetes.io/projected/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-kube-api-access-hnskv\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:36:49.699065 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:49.699028 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82ee94ff-103a-45c8-bdd6-80bb0ed4103e-util\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:36:50.446185 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:50.446156 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" Jul 10 18:36:50.446386 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:50.446155 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afff6g6" event={"ID":"82ee94ff-103a-45c8-bdd6-80bb0ed4103e","Type":"ContainerDied","Data":"ea7345d3e81c98621ac74bf9f2cff4ae75772b4db43e7422f2399cbc9111247c"} Jul 10 18:36:50.446386 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:36:50.446277 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea7345d3e81c98621ac74bf9f2cff4ae75772b4db43e7422f2399cbc9111247c" Jul 10 18:37:00.977873 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:00.977842 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz"] Jul 10 18:37:00.978253 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:00.978062 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ee94ff-103a-45c8-bdd6-80bb0ed4103e" containerName="extract" Jul 10 18:37:00.978253 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:00.978072 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ee94ff-103a-45c8-bdd6-80bb0ed4103e" containerName="extract" Jul 10 18:37:00.978253 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:00.978085 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ee94ff-103a-45c8-bdd6-80bb0ed4103e" containerName="pull" Jul 10 18:37:00.978253 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:00.978090 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ee94ff-103a-45c8-bdd6-80bb0ed4103e" containerName="pull" Jul 10 18:37:00.978253 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:00.978103 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ee94ff-103a-45c8-bdd6-80bb0ed4103e" containerName="util" Jul 10 18:37:00.978253 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:00.978108 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ee94ff-103a-45c8-bdd6-80bb0ed4103e" containerName="util" Jul 10 18:37:00.978253 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:00.978147 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ee94ff-103a-45c8-bdd6-80bb0ed4103e" containerName="extract" Jul 10 18:37:00.985155 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:00.985129 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:00.989691 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:00.989660 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 18:37:00.989832 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:00.989668 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 18:37:00.989832 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:00.989808 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 18:37:00.990534 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:00.990520 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 18:37:00.990623 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:00.990552 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4lmws\"" Jul 10 18:37:00.997250 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:00.997228 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz"] Jul 10 18:37:01.082671 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.082641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bfd1a11-9e2b-4017-8996-699a64b4e0de-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz\" (UID: \"3bfd1a11-9e2b-4017-8996-699a64b4e0de\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.082671 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.082672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bfd1a11-9e2b-4017-8996-699a64b4e0de-tmp\") pod \"opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz\" (UID: \"3bfd1a11-9e2b-4017-8996-699a64b4e0de\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.082866 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.082691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3bfd1a11-9e2b-4017-8996-699a64b4e0de-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz\" (UID: \"3bfd1a11-9e2b-4017-8996-699a64b4e0de\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.082866 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.082763 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3bfd1a11-9e2b-4017-8996-699a64b4e0de-manifests\") pod \"opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz\" (UID: \"3bfd1a11-9e2b-4017-8996-699a64b4e0de\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.082866 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.082802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zldw7\" (UniqueName: \"kubernetes.io/projected/3bfd1a11-9e2b-4017-8996-699a64b4e0de-kube-api-access-zldw7\") pod \"opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz\" (UID: \"3bfd1a11-9e2b-4017-8996-699a64b4e0de\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.183791 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.183755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3bfd1a11-9e2b-4017-8996-699a64b4e0de-manifests\") pod \"opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz\" (UID: \"3bfd1a11-9e2b-4017-8996-699a64b4e0de\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.183791 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.183798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zldw7\" (UniqueName: \"kubernetes.io/projected/3bfd1a11-9e2b-4017-8996-699a64b4e0de-kube-api-access-zldw7\") pod \"opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz\" (UID: \"3bfd1a11-9e2b-4017-8996-699a64b4e0de\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.184018 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.183858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bfd1a11-9e2b-4017-8996-699a64b4e0de-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz\" (UID: \"3bfd1a11-9e2b-4017-8996-699a64b4e0de\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.184018 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.183880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bfd1a11-9e2b-4017-8996-699a64b4e0de-tmp\") pod \"opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz\" (UID: \"3bfd1a11-9e2b-4017-8996-699a64b4e0de\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.184018 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.183909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3bfd1a11-9e2b-4017-8996-699a64b4e0de-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz\" (UID: \"3bfd1a11-9e2b-4017-8996-699a64b4e0de\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.184223 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.184201 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3bfd1a11-9e2b-4017-8996-699a64b4e0de-manifests\") pod \"opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz\" (UID: \"3bfd1a11-9e2b-4017-8996-699a64b4e0de\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.186185 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.186164 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bfd1a11-9e2b-4017-8996-699a64b4e0de-tmp\") pod \"opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz\" (UID: \"3bfd1a11-9e2b-4017-8996-699a64b4e0de\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.186343 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.186321 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3bfd1a11-9e2b-4017-8996-699a64b4e0de-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz\" (UID: \"3bfd1a11-9e2b-4017-8996-699a64b4e0de\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.186402 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.186381 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bfd1a11-9e2b-4017-8996-699a64b4e0de-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz\" (UID: \"3bfd1a11-9e2b-4017-8996-699a64b4e0de\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.194932 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.194908 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zldw7\" (UniqueName: \"kubernetes.io/projected/3bfd1a11-9e2b-4017-8996-699a64b4e0de-kube-api-access-zldw7\") pod \"opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz\" (UID: \"3bfd1a11-9e2b-4017-8996-699a64b4e0de\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.296122 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.296045 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:01.426497 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.426465 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz"] Jul 10 18:37:01.429041 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:37:01.429016 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bfd1a11_9e2b_4017_8996_699a64b4e0de.slice/crio-00ed1e19cc8d52ed3965c7a6dc39b9e2aced37a8de1525184c8d2cbb492bd00c WatchSource:0}: Error finding container 00ed1e19cc8d52ed3965c7a6dc39b9e2aced37a8de1525184c8d2cbb492bd00c: Status 404 returned error can't find the container with id 00ed1e19cc8d52ed3965c7a6dc39b9e2aced37a8de1525184c8d2cbb492bd00c Jul 10 18:37:01.488224 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:01.488188 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" event={"ID":"3bfd1a11-9e2b-4017-8996-699a64b4e0de","Type":"ContainerStarted","Data":"00ed1e19cc8d52ed3965c7a6dc39b9e2aced37a8de1525184c8d2cbb492bd00c"} Jul 10 18:37:05.507585 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:05.507550 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" event={"ID":"3bfd1a11-9e2b-4017-8996-699a64b4e0de","Type":"ContainerStarted","Data":"151bd06cd68ec5aa34e5180125ff6b1415709f94034641632ed9b9f29a855117"} Jul 10 18:37:06.513263 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:06.513226 2576 generic.go:356] "Generic (PLEG): container finished" podID="3bfd1a11-9e2b-4017-8996-699a64b4e0de" containerID="151bd06cd68ec5aa34e5180125ff6b1415709f94034641632ed9b9f29a855117" exitCode=0 Jul 10 18:37:06.513725 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:06.513271 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" event={"ID":"3bfd1a11-9e2b-4017-8996-699a64b4e0de","Type":"ContainerDied","Data":"151bd06cd68ec5aa34e5180125ff6b1415709f94034641632ed9b9f29a855117"} Jul 10 18:37:07.520115 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:07.520080 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" event={"ID":"3bfd1a11-9e2b-4017-8996-699a64b4e0de","Type":"ContainerStarted","Data":"eb686453291af093108e1615855c3c5bb66dd6d20fca9a7f7a2cddbcdc468ec0"} Jul 10 18:37:07.520556 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:07.520176 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:07.541268 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:07.541164 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" podStartSLOduration=2.633931196 podStartE2EDuration="7.541147771s" totalImagesPullingTime="3.869392971s" totalInitContainerRuntime="1.037823604s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:37:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:37:07.540379527 +0000 UTC m=+474.218091991" watchObservedRunningTime="2026-07-10 18:37:07.541147771 +0000 UTC m=+474.218860229" Jul 10 18:37:18.526965 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:18.526932 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz" Jul 10 18:37:21.753393 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.753340 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d88546674-cpbkv"] Jul 10 18:37:21.758404 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.758380 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d88546674-cpbkv" Jul 10 18:37:21.760901 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.760870 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 18:37:21.761763 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.761744 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 18:37:21.761856 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.761746 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 18:37:21.761856 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.761798 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5nns2\"" Jul 10 18:37:21.762445 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.762429 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 18:37:21.776385 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.776346 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d88546674-cpbkv"] Jul 10 18:37:21.802624 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.802595 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-czvzp"] Jul 10 18:37:21.804637 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.804618 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-czvzp" Jul 10 18:37:21.807136 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.807113 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 18:37:21.807554 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.807535 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-fj99l\"" Jul 10 18:37:21.822087 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.822065 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-czvzp"] Jul 10 18:37:21.843647 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.843617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5swkt\" (UniqueName: \"kubernetes.io/projected/7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1-kube-api-access-5swkt\") pod \"kube-auth-proxy-d88546674-cpbkv\" (UID: \"7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1\") " pod="openshift-ingress/kube-auth-proxy-d88546674-cpbkv" Jul 10 18:37:21.843821 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.843702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1-tls-certs\") pod \"kube-auth-proxy-d88546674-cpbkv\" (UID: \"7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1\") " pod="openshift-ingress/kube-auth-proxy-d88546674-cpbkv" Jul 10 18:37:21.843821 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.843737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1-tmp\") pod \"kube-auth-proxy-d88546674-cpbkv\" (UID: \"7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1\") " pod="openshift-ingress/kube-auth-proxy-d88546674-cpbkv" Jul 10 18:37:21.944287 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.944244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5swkt\" (UniqueName: \"kubernetes.io/projected/7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1-kube-api-access-5swkt\") pod \"kube-auth-proxy-d88546674-cpbkv\" (UID: \"7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1\") " pod="openshift-ingress/kube-auth-proxy-d88546674-cpbkv" Jul 10 18:37:21.944492 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.944302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dd66\" (UniqueName: \"kubernetes.io/projected/3be6a366-f512-4c82-887c-195e16cd0ea1-kube-api-access-7dd66\") pod \"odh-model-controller-7998c67dc4-czvzp\" (UID: \"3be6a366-f512-4c82-887c-195e16cd0ea1\") " pod="opendatahub/odh-model-controller-7998c67dc4-czvzp" Jul 10 18:37:21.944492 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.944325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3be6a366-f512-4c82-887c-195e16cd0ea1-cert\") pod \"odh-model-controller-7998c67dc4-czvzp\" (UID: \"3be6a366-f512-4c82-887c-195e16cd0ea1\") " pod="opendatahub/odh-model-controller-7998c67dc4-czvzp" Jul 10 18:37:21.944492 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.944361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1-tls-certs\") pod \"kube-auth-proxy-d88546674-cpbkv\" (UID: \"7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1\") " pod="openshift-ingress/kube-auth-proxy-d88546674-cpbkv" Jul 10 18:37:21.944492 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.944378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1-tmp\") pod \"kube-auth-proxy-d88546674-cpbkv\" (UID: \"7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1\") " pod="openshift-ingress/kube-auth-proxy-d88546674-cpbkv" Jul 10 18:37:21.944492 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:37:21.944479 2576 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 10 18:37:21.944759 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:37:21.944530 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1-tls-certs podName:7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1 nodeName:}" failed. No retries permitted until 2026-07-10 18:37:22.444515151 +0000 UTC m=+489.122227594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1-tls-certs") pod "kube-auth-proxy-d88546674-cpbkv" (UID: "7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1") : secret "kube-auth-proxy-tls" not found Jul 10 18:37:21.946691 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.946667 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1-tmp\") pod \"kube-auth-proxy-d88546674-cpbkv\" (UID: \"7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1\") " pod="openshift-ingress/kube-auth-proxy-d88546674-cpbkv" Jul 10 18:37:21.952613 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:21.952586 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5swkt\" (UniqueName: \"kubernetes.io/projected/7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1-kube-api-access-5swkt\") pod \"kube-auth-proxy-d88546674-cpbkv\" (UID: \"7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1\") " pod="openshift-ingress/kube-auth-proxy-d88546674-cpbkv" Jul 10 18:37:22.045055 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.044967 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dd66\" (UniqueName: \"kubernetes.io/projected/3be6a366-f512-4c82-887c-195e16cd0ea1-kube-api-access-7dd66\") pod \"odh-model-controller-7998c67dc4-czvzp\" (UID: \"3be6a366-f512-4c82-887c-195e16cd0ea1\") " pod="opendatahub/odh-model-controller-7998c67dc4-czvzp" Jul 10 18:37:22.045055 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.045004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3be6a366-f512-4c82-887c-195e16cd0ea1-cert\") pod \"odh-model-controller-7998c67dc4-czvzp\" (UID: \"3be6a366-f512-4c82-887c-195e16cd0ea1\") " pod="opendatahub/odh-model-controller-7998c67dc4-czvzp" Jul 10 18:37:22.045282 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:37:22.045142 2576 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 18:37:22.045282 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:37:22.045192 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be6a366-f512-4c82-887c-195e16cd0ea1-cert podName:3be6a366-f512-4c82-887c-195e16cd0ea1 nodeName:}" failed. No retries permitted until 2026-07-10 18:37:22.545178932 +0000 UTC m=+489.222891372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3be6a366-f512-4c82-887c-195e16cd0ea1-cert") pod "odh-model-controller-7998c67dc4-czvzp" (UID: "3be6a366-f512-4c82-887c-195e16cd0ea1") : secret "odh-model-controller-webhook-cert" not found Jul 10 18:37:22.054654 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.054628 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dd66\" (UniqueName: \"kubernetes.io/projected/3be6a366-f512-4c82-887c-195e16cd0ea1-kube-api-access-7dd66\") pod \"odh-model-controller-7998c67dc4-czvzp\" (UID: \"3be6a366-f512-4c82-887c-195e16cd0ea1\") " pod="opendatahub/odh-model-controller-7998c67dc4-czvzp" Jul 10 18:37:22.397945 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.397911 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zjr4g"] Jul 10 18:37:22.399813 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.399797 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zjr4g" Jul 10 18:37:22.402865 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.402843 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 18:37:22.413829 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.413806 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zjr4g"] Jul 10 18:37:22.448492 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.448457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1-tls-certs\") pod \"kube-auth-proxy-d88546674-cpbkv\" (UID: \"7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1\") " pod="openshift-ingress/kube-auth-proxy-d88546674-cpbkv" Jul 10 18:37:22.450837 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.450803 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1-tls-certs\") pod \"kube-auth-proxy-d88546674-cpbkv\" (UID: \"7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1\") " pod="openshift-ingress/kube-auth-proxy-d88546674-cpbkv" Jul 10 18:37:22.549148 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.549114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3be6a366-f512-4c82-887c-195e16cd0ea1-cert\") pod \"odh-model-controller-7998c67dc4-czvzp\" (UID: \"3be6a366-f512-4c82-887c-195e16cd0ea1\") " pod="opendatahub/odh-model-controller-7998c67dc4-czvzp" Jul 10 18:37:22.549344 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.549168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dfdbad6-5aa2-4443-9398-3a59c736a589-tls-certs\") pod \"model-serving-api-54d89bd5fc-zjr4g\" (UID: \"6dfdbad6-5aa2-4443-9398-3a59c736a589\") " pod="opendatahub/model-serving-api-54d89bd5fc-zjr4g" Jul 10 18:37:22.549344 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.549198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5b42\" (UniqueName: \"kubernetes.io/projected/6dfdbad6-5aa2-4443-9398-3a59c736a589-kube-api-access-g5b42\") pod \"model-serving-api-54d89bd5fc-zjr4g\" (UID: \"6dfdbad6-5aa2-4443-9398-3a59c736a589\") " pod="opendatahub/model-serving-api-54d89bd5fc-zjr4g" Jul 10 18:37:22.551537 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.551506 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3be6a366-f512-4c82-887c-195e16cd0ea1-cert\") pod \"odh-model-controller-7998c67dc4-czvzp\" (UID: \"3be6a366-f512-4c82-887c-195e16cd0ea1\") " pod="opendatahub/odh-model-controller-7998c67dc4-czvzp" Jul 10 18:37:22.650000 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.649918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dfdbad6-5aa2-4443-9398-3a59c736a589-tls-certs\") pod \"model-serving-api-54d89bd5fc-zjr4g\" (UID: \"6dfdbad6-5aa2-4443-9398-3a59c736a589\") " pod="opendatahub/model-serving-api-54d89bd5fc-zjr4g" Jul 10 18:37:22.650000 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.649951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5b42\" (UniqueName: \"kubernetes.io/projected/6dfdbad6-5aa2-4443-9398-3a59c736a589-kube-api-access-g5b42\") pod \"model-serving-api-54d89bd5fc-zjr4g\" (UID: \"6dfdbad6-5aa2-4443-9398-3a59c736a589\") " pod="opendatahub/model-serving-api-54d89bd5fc-zjr4g" Jul 10 18:37:22.652540 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.652511 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dfdbad6-5aa2-4443-9398-3a59c736a589-tls-certs\") pod \"model-serving-api-54d89bd5fc-zjr4g\" (UID: \"6dfdbad6-5aa2-4443-9398-3a59c736a589\") " pod="opendatahub/model-serving-api-54d89bd5fc-zjr4g" Jul 10 18:37:22.660922 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.660890 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5b42\" (UniqueName: \"kubernetes.io/projected/6dfdbad6-5aa2-4443-9398-3a59c736a589-kube-api-access-g5b42\") pod \"model-serving-api-54d89bd5fc-zjr4g\" (UID: \"6dfdbad6-5aa2-4443-9398-3a59c736a589\") " pod="opendatahub/model-serving-api-54d89bd5fc-zjr4g" Jul 10 18:37:22.667611 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.667588 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d88546674-cpbkv" Jul 10 18:37:22.709571 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.709540 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zjr4g" Jul 10 18:37:22.716639 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.716597 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-czvzp" Jul 10 18:37:22.828057 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:37:22.828021 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7e0f57_d7a2_428d_8dd1_3562d6ab25b1.slice/crio-0cd7de875be048c91a922d55653e8d753a860092e59ac40cea79754392019d69 WatchSource:0}: Error finding container 0cd7de875be048c91a922d55653e8d753a860092e59ac40cea79754392019d69: Status 404 returned error can't find the container with id 0cd7de875be048c91a922d55653e8d753a860092e59ac40cea79754392019d69 Jul 10 18:37:22.828382 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.828079 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d88546674-cpbkv"] Jul 10 18:37:22.869060 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.869030 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-czvzp"] Jul 10 18:37:22.871661 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:37:22.871632 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be6a366_f512_4c82_887c_195e16cd0ea1.slice/crio-4cdadaedda21ebf0dfacba0c0fd635848314859b0ba32fab2a81f9acfc9daad0 WatchSource:0}: Error finding container 4cdadaedda21ebf0dfacba0c0fd635848314859b0ba32fab2a81f9acfc9daad0: Status 404 returned error can't find the container with id 4cdadaedda21ebf0dfacba0c0fd635848314859b0ba32fab2a81f9acfc9daad0 Jul 10 18:37:22.890603 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:22.890550 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zjr4g"] Jul 10 18:37:22.892416 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:37:22.892394 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfdbad6_5aa2_4443_9398_3a59c736a589.slice/crio-df17a241ebb2fc1e87a20132a92b62072540e45d53ead8c676c5f09c3fb00fce WatchSource:0}: Error finding container df17a241ebb2fc1e87a20132a92b62072540e45d53ead8c676c5f09c3fb00fce: Status 404 returned error can't find the container with id df17a241ebb2fc1e87a20132a92b62072540e45d53ead8c676c5f09c3fb00fce Jul 10 18:37:23.052111 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.052025 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f7flk"] Jul 10 18:37:23.054873 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.054855 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.057206 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.057181 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 18:37:23.057206 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.057195 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 18:37:23.057652 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.057600 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h7wqg\"" Jul 10 18:37:23.057777 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.057665 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 18:37:23.057835 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.057779 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 18:37:23.067074 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.067052 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f7flk"] Jul 10 18:37:23.154743 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.154712 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-local-certs\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.154916 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.154757 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7z52\" (UniqueName: \"kubernetes.io/projected/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-kube-api-access-b7z52\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.154916 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.154781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-token\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.154916 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.154880 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.155009 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.154924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.155009 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.154994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.155073 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.155029 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-cacerts\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.256020 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.255990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-cacerts\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.256219 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.256037 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-local-certs\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.256219 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.256079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7z52\" (UniqueName: \"kubernetes.io/projected/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-kube-api-access-b7z52\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.256219 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.256108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-token\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.256219 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.256139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.256219 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.256166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.256499 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.256347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.257037 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.256981 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.259037 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.259003 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-cacerts\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.259147 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.259047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.259147 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.259075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-local-certs\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.259147 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.259124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.268956 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.268928 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-token\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.269259 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.269236 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7z52\" (UniqueName: \"kubernetes.io/projected/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-kube-api-access-b7z52\") pod \"istiod-openshift-gateway-7574459656-f7flk\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.365736 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.365700 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:23.516901 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.516789 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f7flk"] Jul 10 18:37:23.519524 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:37:23.519494 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b003c5_75b9_4bdc_bc63_f4cd3d9d6f3f.slice/crio-454f404ddc98d721ae4bfe3c962d7adb42ed10ff31a4847bcf54f609c25744de WatchSource:0}: Error finding container 454f404ddc98d721ae4bfe3c962d7adb42ed10ff31a4847bcf54f609c25744de: Status 404 returned error can't find the container with id 454f404ddc98d721ae4bfe3c962d7adb42ed10ff31a4847bcf54f609c25744de Jul 10 18:37:23.591635 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.591529 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-czvzp" event={"ID":"3be6a366-f512-4c82-887c-195e16cd0ea1","Type":"ContainerStarted","Data":"4cdadaedda21ebf0dfacba0c0fd635848314859b0ba32fab2a81f9acfc9daad0"} Jul 10 18:37:23.593788 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.593715 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d88546674-cpbkv" event={"ID":"7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1","Type":"ContainerStarted","Data":"0cd7de875be048c91a922d55653e8d753a860092e59ac40cea79754392019d69"} Jul 10 18:37:23.596285 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.596212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zjr4g" event={"ID":"6dfdbad6-5aa2-4443-9398-3a59c736a589","Type":"ContainerStarted","Data":"df17a241ebb2fc1e87a20132a92b62072540e45d53ead8c676c5f09c3fb00fce"} Jul 10 18:37:23.598015 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:23.597988 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" event={"ID":"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f","Type":"ContainerStarted","Data":"454f404ddc98d721ae4bfe3c962d7adb42ed10ff31a4847bcf54f609c25744de"} Jul 10 18:37:27.617745 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:27.617711 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-czvzp" event={"ID":"3be6a366-f512-4c82-887c-195e16cd0ea1","Type":"ContainerStarted","Data":"cc9323a662014a9753d60469a73694cc3ee62fe090b8bebd3fe80a8dab3d97e2"} Jul 10 18:37:27.618204 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:27.617834 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-czvzp" Jul 10 18:37:27.619239 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:27.619213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d88546674-cpbkv" event={"ID":"7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1","Type":"ContainerStarted","Data":"5f71aaa9b3eebad0d05b3d7e64ac89676749fe32ee909db81096b88dbaf03e92"} Jul 10 18:37:27.620536 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:27.620514 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zjr4g" event={"ID":"6dfdbad6-5aa2-4443-9398-3a59c736a589","Type":"ContainerStarted","Data":"10885eda1b15c3cab93ae03ee911f89c0e9abbec986785cc1afa24619159cce3"} Jul 10 18:37:27.620617 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:27.620566 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-zjr4g" Jul 10 18:37:27.635277 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:27.635236 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-czvzp" podStartSLOduration=2.641237559 podStartE2EDuration="6.635225532s" totalImagesPullingTime="3.993987973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:37:27.634021482 +0000 UTC m=+494.311733944" watchObservedRunningTime="2026-07-10 18:37:27.635225532 +0000 UTC m=+494.312937971" Jul 10 18:37:27.665248 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:27.665205 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-zjr4g" podStartSLOduration=1.713865918 podStartE2EDuration="5.665191316s" totalImagesPullingTime="3.951325398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:37:27.663204083 +0000 UTC m=+494.340916544" watchObservedRunningTime="2026-07-10 18:37:27.665191316 +0000 UTC m=+494.342903778" Jul 10 18:37:27.693552 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:27.693496 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d88546674-cpbkv" podStartSLOduration=2.669854892 podStartE2EDuration="6.693481595s" totalImagesPullingTime="4.023626703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:37:27.691298865 +0000 UTC m=+494.369011328" watchObservedRunningTime="2026-07-10 18:37:27.693481595 +0000 UTC m=+494.371194056" Jul 10 18:37:32.845799 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:32.845765 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g"] Jul 10 18:37:32.848441 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:32.848424 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g" Jul 10 18:37:32.851161 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:32.851143 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 10 18:37:32.851281 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:32.851200 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9z8xq\"" Jul 10 18:37:32.853844 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:32.853822 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc"] Jul 10 18:37:32.857368 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:32.857331 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc" Jul 10 18:37:32.859817 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:32.859691 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-4rcn6\"" Jul 10 18:37:32.860558 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:32.860536 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 10 18:37:32.861313 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:32.861291 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g"] Jul 10 18:37:32.871154 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:32.871132 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc"] Jul 10 18:37:33.043657 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.043624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9600ebbe-f0bc-4265-884b-274efae32892-cert\") pod \"kserve-controller-manager-d6f4b65b8-hfj7g\" (UID: \"9600ebbe-f0bc-4265-884b-274efae32892\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g" Jul 10 18:37:33.043657 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.043668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79685dda-acd4-4446-93b1-2608b69dd463-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wkrqc\" (UID: \"79685dda-acd4-4446-93b1-2608b69dd463\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc" Jul 10 18:37:33.043878 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.043744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht5l4\" (UniqueName: \"kubernetes.io/projected/9600ebbe-f0bc-4265-884b-274efae32892-kube-api-access-ht5l4\") pod \"kserve-controller-manager-d6f4b65b8-hfj7g\" (UID: \"9600ebbe-f0bc-4265-884b-274efae32892\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g" Jul 10 18:37:33.043878 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.043814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm69d\" (UniqueName: \"kubernetes.io/projected/79685dda-acd4-4446-93b1-2608b69dd463-kube-api-access-sm69d\") pod \"llmisvc-controller-manager-777c6bd7d5-wkrqc\" (UID: \"79685dda-acd4-4446-93b1-2608b69dd463\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc" Jul 10 18:37:33.144422 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.144324 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht5l4\" (UniqueName: \"kubernetes.io/projected/9600ebbe-f0bc-4265-884b-274efae32892-kube-api-access-ht5l4\") pod \"kserve-controller-manager-d6f4b65b8-hfj7g\" (UID: \"9600ebbe-f0bc-4265-884b-274efae32892\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g" Jul 10 18:37:33.144422 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.144387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm69d\" (UniqueName: \"kubernetes.io/projected/79685dda-acd4-4446-93b1-2608b69dd463-kube-api-access-sm69d\") pod \"llmisvc-controller-manager-777c6bd7d5-wkrqc\" (UID: \"79685dda-acd4-4446-93b1-2608b69dd463\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc" Jul 10 18:37:33.144422 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.144420 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9600ebbe-f0bc-4265-884b-274efae32892-cert\") pod \"kserve-controller-manager-d6f4b65b8-hfj7g\" (UID: \"9600ebbe-f0bc-4265-884b-274efae32892\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g" Jul 10 18:37:33.144644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.144437 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79685dda-acd4-4446-93b1-2608b69dd463-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wkrqc\" (UID: \"79685dda-acd4-4446-93b1-2608b69dd463\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc" Jul 10 18:37:33.144644 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:37:33.144546 2576 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 10 18:37:33.144644 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:37:33.144546 2576 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 10 18:37:33.144644 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:37:33.144599 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9600ebbe-f0bc-4265-884b-274efae32892-cert podName:9600ebbe-f0bc-4265-884b-274efae32892 nodeName:}" failed. No retries permitted until 2026-07-10 18:37:33.64458368 +0000 UTC m=+500.322296125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9600ebbe-f0bc-4265-884b-274efae32892-cert") pod "kserve-controller-manager-d6f4b65b8-hfj7g" (UID: "9600ebbe-f0bc-4265-884b-274efae32892") : secret "kserve-webhook-server-cert" not found Jul 10 18:37:33.144785 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:37:33.144666 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79685dda-acd4-4446-93b1-2608b69dd463-cert podName:79685dda-acd4-4446-93b1-2608b69dd463 nodeName:}" failed. No retries permitted until 2026-07-10 18:37:33.644650713 +0000 UTC m=+500.322363156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/79685dda-acd4-4446-93b1-2608b69dd463-cert") pod "llmisvc-controller-manager-777c6bd7d5-wkrqc" (UID: "79685dda-acd4-4446-93b1-2608b69dd463") : secret "llmisvc-webhook-server-cert" not found Jul 10 18:37:33.153335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.153315 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm69d\" (UniqueName: \"kubernetes.io/projected/79685dda-acd4-4446-93b1-2608b69dd463-kube-api-access-sm69d\") pod \"llmisvc-controller-manager-777c6bd7d5-wkrqc\" (UID: \"79685dda-acd4-4446-93b1-2608b69dd463\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc" Jul 10 18:37:33.155671 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.155645 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht5l4\" (UniqueName: \"kubernetes.io/projected/9600ebbe-f0bc-4265-884b-274efae32892-kube-api-access-ht5l4\") pod \"kserve-controller-manager-d6f4b65b8-hfj7g\" (UID: \"9600ebbe-f0bc-4265-884b-274efae32892\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g" Jul 10 18:37:33.648918 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.648887 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9600ebbe-f0bc-4265-884b-274efae32892-cert\") pod \"kserve-controller-manager-d6f4b65b8-hfj7g\" (UID: \"9600ebbe-f0bc-4265-884b-274efae32892\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g" Jul 10 18:37:33.648918 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.648922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79685dda-acd4-4446-93b1-2608b69dd463-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wkrqc\" (UID: \"79685dda-acd4-4446-93b1-2608b69dd463\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc" Jul 10 18:37:33.651714 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.651687 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79685dda-acd4-4446-93b1-2608b69dd463-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wkrqc\" (UID: \"79685dda-acd4-4446-93b1-2608b69dd463\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc" Jul 10 18:37:33.651847 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.651739 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9600ebbe-f0bc-4265-884b-274efae32892-cert\") pod \"kserve-controller-manager-d6f4b65b8-hfj7g\" (UID: \"9600ebbe-f0bc-4265-884b-274efae32892\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g" Jul 10 18:37:33.760881 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.760841 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g" Jul 10 18:37:33.770059 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:33.770021 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc" Jul 10 18:37:34.473638 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:34.473568 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g"] Jul 10 18:37:34.493872 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:34.493851 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc"] Jul 10 18:37:34.543784 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:37:34.543729 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9600ebbe_f0bc_4265_884b_274efae32892.slice/crio-a4d41a8748aa8cfd3128e30a07406abb72060ba3a272dca64d5740bce98a34f4 WatchSource:0}: Error finding container a4d41a8748aa8cfd3128e30a07406abb72060ba3a272dca64d5740bce98a34f4: Status 404 returned error can't find the container with id a4d41a8748aa8cfd3128e30a07406abb72060ba3a272dca64d5740bce98a34f4 Jul 10 18:37:34.544666 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:37:34.544636 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod79685dda_acd4_4446_93b1_2608b69dd463.slice/crio-0cb418c0e22ee89b3c142fe53ef6a902cf69cc92840769421cf90ca8687a857d WatchSource:0}: Error finding container 0cb418c0e22ee89b3c142fe53ef6a902cf69cc92840769421cf90ca8687a857d: Status 404 returned error can't find the container with id 0cb418c0e22ee89b3c142fe53ef6a902cf69cc92840769421cf90ca8687a857d Jul 10 18:37:34.560173 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:34.560130 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:37:34.560256 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:34.560224 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:37:34.650370 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:34.650322 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g" event={"ID":"9600ebbe-f0bc-4265-884b-274efae32892","Type":"ContainerStarted","Data":"a4d41a8748aa8cfd3128e30a07406abb72060ba3a272dca64d5740bce98a34f4"} Jul 10 18:37:34.651577 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:34.651547 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc" event={"ID":"79685dda-acd4-4446-93b1-2608b69dd463","Type":"ContainerStarted","Data":"0cb418c0e22ee89b3c142fe53ef6a902cf69cc92840769421cf90ca8687a857d"} Jul 10 18:37:35.658631 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:35.658597 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" event={"ID":"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f","Type":"ContainerStarted","Data":"14bbf3585e42f69d2773ccc59c07c0f229d4fea00cfa09c74d7ff9a34f46c6c7"} Jul 10 18:37:35.659078 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:35.658939 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:35.660653 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:35.660626 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-f7flk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 18:37:35.660765 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:35.660671 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" podUID="37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 18:37:35.693853 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:35.693788 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" podStartSLOduration=1.6560276809999999 podStartE2EDuration="12.693772498s" totalImagesPullingTime="11.037744817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:37:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:37:35.692903293 +0000 UTC m=+502.370615755" watchObservedRunningTime="2026-07-10 18:37:35.693772498 +0000 UTC m=+502.371484960" Jul 10 18:37:35.966117 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:35.966043 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g"] Jul 10 18:37:35.970132 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:35.970066 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:35.972849 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:35.972827 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-wzvsq\"" Jul 10 18:37:35.985970 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:35.985908 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g"] Jul 10 18:37:36.066421 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.066338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2af4c51-c115-4f25-8868-4ab579f9633a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.066421 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.066401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2af4c51-c115-4f25-8868-4ab579f9633a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.066679 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.066439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2af4c51-c115-4f25-8868-4ab579f9633a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.066679 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.066459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2af4c51-c115-4f25-8868-4ab579f9633a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.066679 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.066485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48wvp\" (UniqueName: \"kubernetes.io/projected/e2af4c51-c115-4f25-8868-4ab579f9633a-kube-api-access-48wvp\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.066679 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.066513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2af4c51-c115-4f25-8868-4ab579f9633a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.066679 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.066605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2af4c51-c115-4f25-8868-4ab579f9633a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.066679 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.066651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2af4c51-c115-4f25-8868-4ab579f9633a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.066881 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.066695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2af4c51-c115-4f25-8868-4ab579f9633a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.167416 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.167375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2af4c51-c115-4f25-8868-4ab579f9633a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.167591 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.167448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48wvp\" (UniqueName: \"kubernetes.io/projected/e2af4c51-c115-4f25-8868-4ab579f9633a-kube-api-access-48wvp\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.167591 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.167493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2af4c51-c115-4f25-8868-4ab579f9633a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.167591 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.167522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2af4c51-c115-4f25-8868-4ab579f9633a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.167591 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.167547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2af4c51-c115-4f25-8868-4ab579f9633a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.167591 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.167581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2af4c51-c115-4f25-8868-4ab579f9633a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.167852 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.167630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2af4c51-c115-4f25-8868-4ab579f9633a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.167852 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.167656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2af4c51-c115-4f25-8868-4ab579f9633a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.167852 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.167729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2af4c51-c115-4f25-8868-4ab579f9633a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.167852 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.167771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2af4c51-c115-4f25-8868-4ab579f9633a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.168118 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.168067 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2af4c51-c115-4f25-8868-4ab579f9633a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.168412 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.168388 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2af4c51-c115-4f25-8868-4ab579f9633a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.168504 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.168423 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2af4c51-c115-4f25-8868-4ab579f9633a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.168788 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.168765 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2af4c51-c115-4f25-8868-4ab579f9633a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.170461 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.170440 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2af4c51-c115-4f25-8868-4ab579f9633a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.170943 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.170920 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2af4c51-c115-4f25-8868-4ab579f9633a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.179836 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.179808 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2af4c51-c115-4f25-8868-4ab579f9633a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.180084 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.180052 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48wvp\" (UniqueName: \"kubernetes.io/projected/e2af4c51-c115-4f25-8868-4ab579f9633a-kube-api-access-48wvp\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6787xd5g\" (UID: \"e2af4c51-c115-4f25-8868-4ab579f9633a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.301285 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.301189 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:36.455374 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.455326 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g"] Jul 10 18:37:36.457031 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:37:36.457004 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2af4c51_c115_4f25_8868_4ab579f9633a.slice/crio-b6b1f9aa31b2008fab0183e70374db736b712aabb5022e3b3c7de7ccc10be121 WatchSource:0}: Error finding container b6b1f9aa31b2008fab0183e70374db736b712aabb5022e3b3c7de7ccc10be121: Status 404 returned error can't find the container with id b6b1f9aa31b2008fab0183e70374db736b712aabb5022e3b3c7de7ccc10be121 Jul 10 18:37:36.665026 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.664979 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" event={"ID":"e2af4c51-c115-4f25-8868-4ab579f9633a","Type":"ContainerStarted","Data":"b6b1f9aa31b2008fab0183e70374db736b712aabb5022e3b3c7de7ccc10be121"} Jul 10 18:37:36.665921 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:36.665899 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:38.627015 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:38.626972 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-czvzp" Jul 10 18:37:38.628743 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:38.628726 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-zjr4g" Jul 10 18:37:38.678464 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:38.677690 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g" event={"ID":"9600ebbe-f0bc-4265-884b-274efae32892","Type":"ContainerStarted","Data":"5eeff984836186d2533b92432c360ea2881d9f3693e775e54991d84a624ffadf"} Jul 10 18:37:38.678464 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:38.678443 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g" Jul 10 18:37:38.681407 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:38.680991 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc" event={"ID":"79685dda-acd4-4446-93b1-2608b69dd463","Type":"ContainerStarted","Data":"d5be3c25110683c82c8bb65311a2574187805c5bf6fd65a0f06659a971ca0a35"} Jul 10 18:37:38.681407 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:38.681049 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc" Jul 10 18:37:38.696870 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:38.696810 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g" podStartSLOduration=3.2182777 podStartE2EDuration="6.696789383s" totalImagesPullingTime="3.478511683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:37:38.694668746 +0000 UTC m=+505.372381212" watchObservedRunningTime="2026-07-10 18:37:38.696789383 +0000 UTC m=+505.374501864" Jul 10 18:37:38.710491 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:38.710439 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc" podStartSLOduration=3.226025762 podStartE2EDuration="6.710420862s" totalImagesPullingTime="3.4843951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:37:38.709712188 +0000 UTC m=+505.387424652" watchObservedRunningTime="2026-07-10 18:37:38.710420862 +0000 UTC m=+505.388133327" Jul 10 18:37:40.367029 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:40.366993 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:37:40.367292 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:40.367067 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:37:40.367292 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:40.367097 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:37:40.683619 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:40.683536 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f7flk"] Jul 10 18:37:40.683773 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:40.683719 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" podUID="37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f" containerName="discovery" containerID="cri-o://14bbf3585e42f69d2773ccc59c07c0f229d4fea00cfa09c74d7ff9a34f46c6c7" gracePeriod=30 Jul 10 18:37:40.693545 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:40.693514 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" event={"ID":"e2af4c51-c115-4f25-8868-4ab579f9633a","Type":"ContainerStarted","Data":"ed746b048ee35817af20fd884dee2b1ea4a6d6d79171a9aa014cf1a23f4a003d"} Jul 10 18:37:40.726317 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:40.726267 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" podStartSLOduration=1.818598688 podStartE2EDuration="5.726247302s" totalImagesPullingTime="3.907648614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:37:40.725212188 +0000 UTC m=+507.402924650" watchObservedRunningTime="2026-07-10 18:37:40.726247302 +0000 UTC m=+507.403959761" Jul 10 18:37:40.934985 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:40.934923 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:41.008784 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.008755 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-csr-ca-configmap\") pod \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " Jul 10 18:37:41.008943 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.008793 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-kubeconfig\") pod \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " Jul 10 18:37:41.008943 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.008818 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7z52\" (UniqueName: \"kubernetes.io/projected/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-kube-api-access-b7z52\") pod \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " Jul 10 18:37:41.008943 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.008853 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-cacerts\") pod \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " Jul 10 18:37:41.008943 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.008893 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-token\") pod \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " Jul 10 18:37:41.008943 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.008911 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-local-certs\") pod \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " Jul 10 18:37:41.008943 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.008939 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-csr-dns-cert\") pod \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\" (UID: \"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f\") " Jul 10 18:37:41.009253 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.009173 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f" (UID: "37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:37:41.011251 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.011215 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-cacerts" (OuterVolumeSpecName: "cacerts") pod "37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f" (UID: "37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:37:41.011251 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.011228 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-token" (OuterVolumeSpecName: "istio-token") pod "37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f" (UID: "37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:37:41.011503 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.011474 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f" (UID: "37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:37:41.011503 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.011495 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-kube-api-access-b7z52" (OuterVolumeSpecName: "kube-api-access-b7z52") pod "37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f" (UID: "37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f"). InnerVolumeSpecName "kube-api-access-b7z52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:37:41.011629 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.011549 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f" (UID: "37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:37:41.011629 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.011565 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-local-certs" (OuterVolumeSpecName: "local-certs") pod "37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f" (UID: "37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:37:41.109475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.109444 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7z52\" (UniqueName: \"kubernetes.io/projected/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-kube-api-access-b7z52\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:37:41.109475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.109469 2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-cacerts\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:37:41.109475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.109479 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-token\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:37:41.109688 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.109488 2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-local-certs\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:37:41.109688 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.109496 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-csr-dns-cert\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:37:41.109688 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.109503 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-csr-ca-configmap\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:37:41.109688 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.109512 2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f-istio-kubeconfig\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:37:41.301505 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.301425 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:41.305789 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.305765 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:41.699386 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.699332 2576 generic.go:356] "Generic (PLEG): container finished" podID="37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f" containerID="14bbf3585e42f69d2773ccc59c07c0f229d4fea00cfa09c74d7ff9a34f46c6c7" exitCode=0 Jul 10 18:37:41.699781 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.699398 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" Jul 10 18:37:41.699781 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.699415 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" event={"ID":"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f","Type":"ContainerDied","Data":"14bbf3585e42f69d2773ccc59c07c0f229d4fea00cfa09c74d7ff9a34f46c6c7"} Jul 10 18:37:41.699781 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.699462 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f7flk" event={"ID":"37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f","Type":"ContainerDied","Data":"454f404ddc98d721ae4bfe3c962d7adb42ed10ff31a4847bcf54f609c25744de"} Jul 10 18:37:41.699781 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.699482 2576 scope.go:117] "RemoveContainer" containerID="14bbf3585e42f69d2773ccc59c07c0f229d4fea00cfa09c74d7ff9a34f46c6c7" Jul 10 18:37:41.699981 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.699829 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:41.700886 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.700806 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6787xd5g" Jul 10 18:37:41.709052 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.709036 2576 scope.go:117] "RemoveContainer" containerID="14bbf3585e42f69d2773ccc59c07c0f229d4fea00cfa09c74d7ff9a34f46c6c7" Jul 10 18:37:41.709289 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:37:41.709271 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14bbf3585e42f69d2773ccc59c07c0f229d4fea00cfa09c74d7ff9a34f46c6c7\": container with ID starting with 14bbf3585e42f69d2773ccc59c07c0f229d4fea00cfa09c74d7ff9a34f46c6c7 not found: ID does not exist" containerID="14bbf3585e42f69d2773ccc59c07c0f229d4fea00cfa09c74d7ff9a34f46c6c7" Jul 10 18:37:41.709339 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.709295 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14bbf3585e42f69d2773ccc59c07c0f229d4fea00cfa09c74d7ff9a34f46c6c7"} err="failed to get container status \"14bbf3585e42f69d2773ccc59c07c0f229d4fea00cfa09c74d7ff9a34f46c6c7\": rpc error: code = NotFound desc = could not find container \"14bbf3585e42f69d2773ccc59c07c0f229d4fea00cfa09c74d7ff9a34f46c6c7\": container with ID starting with 14bbf3585e42f69d2773ccc59c07c0f229d4fea00cfa09c74d7ff9a34f46c6c7 not found: ID does not exist" Jul 10 18:37:41.741804 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.741776 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f7flk"] Jul 10 18:37:41.749057 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.749026 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f7flk"] Jul 10 18:37:41.843181 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:37:41.843149 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f" path="/var/lib/kubelet/pods/37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f/volumes" Jul 10 18:38:09.688456 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:09.688384 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wkrqc" Jul 10 18:38:10.699070 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:10.699039 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hfj7g" Jul 10 18:38:22.483473 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:22.483440 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pfqvt"] Jul 10 18:38:22.483908 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:22.483773 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f" containerName="discovery" Jul 10 18:38:22.483908 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:22.483786 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f" containerName="discovery" Jul 10 18:38:22.483908 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:22.483868 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b003c5-75b9-4bdc-bc63-f4cd3d9d6f3f" containerName="discovery" Jul 10 18:38:22.492038 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:22.492009 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pfqvt" Jul 10 18:38:22.494655 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:22.494630 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 18:38:22.494824 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:22.494661 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 18:38:22.494905 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:22.494684 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-vz4jx\"" Jul 10 18:38:22.498469 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:22.498447 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pfqvt"] Jul 10 18:38:22.650293 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:22.650264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk4zs\" (UniqueName: \"kubernetes.io/projected/bf7b4339-b003-414f-a5d8-64c79be04e48-kube-api-access-xk4zs\") pod \"kuadrant-operator-catalog-pfqvt\" (UID: \"bf7b4339-b003-414f-a5d8-64c79be04e48\") " pod="kuadrant-system/kuadrant-operator-catalog-pfqvt" Jul 10 18:38:22.751048 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:22.750971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk4zs\" (UniqueName: \"kubernetes.io/projected/bf7b4339-b003-414f-a5d8-64c79be04e48-kube-api-access-xk4zs\") pod \"kuadrant-operator-catalog-pfqvt\" (UID: \"bf7b4339-b003-414f-a5d8-64c79be04e48\") " pod="kuadrant-system/kuadrant-operator-catalog-pfqvt" Jul 10 18:38:22.758808 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:22.758781 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk4zs\" (UniqueName: \"kubernetes.io/projected/bf7b4339-b003-414f-a5d8-64c79be04e48-kube-api-access-xk4zs\") pod \"kuadrant-operator-catalog-pfqvt\" (UID: \"bf7b4339-b003-414f-a5d8-64c79be04e48\") " pod="kuadrant-system/kuadrant-operator-catalog-pfqvt" Jul 10 18:38:22.804415 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:22.804382 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pfqvt" Jul 10 18:38:22.840440 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:22.840401 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pfqvt"] Jul 10 18:38:22.931121 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:22.931092 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pfqvt"] Jul 10 18:38:22.934215 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:38:22.934175 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7b4339_b003_414f_a5d8_64c79be04e48.slice/crio-6ef926fd0dc57d9b0efe39a1c694264dea4dec8aa8671c22d99ee59bb1ee68e9 WatchSource:0}: Error finding container 6ef926fd0dc57d9b0efe39a1c694264dea4dec8aa8671c22d99ee59bb1ee68e9: Status 404 returned error can't find the container with id 6ef926fd0dc57d9b0efe39a1c694264dea4dec8aa8671c22d99ee59bb1ee68e9 Jul 10 18:38:23.053949 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:23.053869 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tbc99"] Jul 10 18:38:23.056423 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:23.056408 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tbc99" Jul 10 18:38:23.066152 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:23.066130 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tbc99"] Jul 10 18:38:23.155301 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:23.155269 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55rbm\" (UniqueName: \"kubernetes.io/projected/353eda98-7872-455e-9f82-e36f7df6d1c3-kube-api-access-55rbm\") pod \"kuadrant-operator-catalog-tbc99\" (UID: \"353eda98-7872-455e-9f82-e36f7df6d1c3\") " pod="kuadrant-system/kuadrant-operator-catalog-tbc99" Jul 10 18:38:23.256627 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:23.256590 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55rbm\" (UniqueName: \"kubernetes.io/projected/353eda98-7872-455e-9f82-e36f7df6d1c3-kube-api-access-55rbm\") pod \"kuadrant-operator-catalog-tbc99\" (UID: \"353eda98-7872-455e-9f82-e36f7df6d1c3\") " pod="kuadrant-system/kuadrant-operator-catalog-tbc99" Jul 10 18:38:23.264261 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:23.264235 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55rbm\" (UniqueName: \"kubernetes.io/projected/353eda98-7872-455e-9f82-e36f7df6d1c3-kube-api-access-55rbm\") pod \"kuadrant-operator-catalog-tbc99\" (UID: \"353eda98-7872-455e-9f82-e36f7df6d1c3\") " pod="kuadrant-system/kuadrant-operator-catalog-tbc99" Jul 10 18:38:23.365061 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:23.365027 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tbc99" Jul 10 18:38:23.490970 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:23.490939 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tbc99"] Jul 10 18:38:23.493450 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:38:23.493424 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353eda98_7872_455e_9f82_e36f7df6d1c3.slice/crio-8450c78c3c6c4553701ac87f95ef553de169160afdab1aa955673abcf138e60a WatchSource:0}: Error finding container 8450c78c3c6c4553701ac87f95ef553de169160afdab1aa955673abcf138e60a: Status 404 returned error can't find the container with id 8450c78c3c6c4553701ac87f95ef553de169160afdab1aa955673abcf138e60a Jul 10 18:38:23.882974 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:23.882925 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tbc99" event={"ID":"353eda98-7872-455e-9f82-e36f7df6d1c3","Type":"ContainerStarted","Data":"8450c78c3c6c4553701ac87f95ef553de169160afdab1aa955673abcf138e60a"} Jul 10 18:38:23.884379 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:23.884317 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pfqvt" event={"ID":"bf7b4339-b003-414f-a5d8-64c79be04e48","Type":"ContainerStarted","Data":"6ef926fd0dc57d9b0efe39a1c694264dea4dec8aa8671c22d99ee59bb1ee68e9"} Jul 10 18:38:24.892380 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:24.892332 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tbc99" event={"ID":"353eda98-7872-455e-9f82-e36f7df6d1c3","Type":"ContainerStarted","Data":"dd4e5ac60358e05f3e4d6434a82aeaa54006e10effcd31b68d140ff1a58214db"} Jul 10 18:38:24.893994 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:24.893969 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pfqvt" event={"ID":"bf7b4339-b003-414f-a5d8-64c79be04e48","Type":"ContainerStarted","Data":"cc52967b04da4e8aa832a2ea0c7161c7e21292a54a0b50f35e98bd8577c0972e"} Jul 10 18:38:24.894129 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:24.894044 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-pfqvt" podUID="bf7b4339-b003-414f-a5d8-64c79be04e48" containerName="registry-server" containerID="cri-o://cc52967b04da4e8aa832a2ea0c7161c7e21292a54a0b50f35e98bd8577c0972e" gracePeriod=2 Jul 10 18:38:24.909530 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:24.909476 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-tbc99" podStartSLOduration=0.604367281 podStartE2EDuration="1.90945679s" totalImagesPullingTime="1.305089509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:38:24.907566101 +0000 UTC m=+551.585278563" watchObservedRunningTime="2026-07-10 18:38:24.90945679 +0000 UTC m=+551.587169253" Jul 10 18:38:24.922402 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:24.922326 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-pfqvt" podStartSLOduration=1.054175935 podStartE2EDuration="2.922306376s" totalImagesPullingTime="1.868130441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:38:24.92102376 +0000 UTC m=+551.598736223" watchObservedRunningTime="2026-07-10 18:38:24.922306376 +0000 UTC m=+551.600018840" Jul 10 18:38:25.136006 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:25.135982 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-pfqvt_bf7b4339-b003-414f-a5d8-64c79be04e48/registry-server/0.log" Jul 10 18:38:25.137088 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:25.137073 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pfqvt" Jul 10 18:38:25.273551 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:25.273520 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk4zs\" (UniqueName: \"kubernetes.io/projected/bf7b4339-b003-414f-a5d8-64c79be04e48-kube-api-access-xk4zs\") pod \"bf7b4339-b003-414f-a5d8-64c79be04e48\" (UID: \"bf7b4339-b003-414f-a5d8-64c79be04e48\") " Jul 10 18:38:25.275650 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:25.275619 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7b4339-b003-414f-a5d8-64c79be04e48-kube-api-access-xk4zs" (OuterVolumeSpecName: "kube-api-access-xk4zs") pod "bf7b4339-b003-414f-a5d8-64c79be04e48" (UID: "bf7b4339-b003-414f-a5d8-64c79be04e48"). InnerVolumeSpecName "kube-api-access-xk4zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:38:25.374290 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:25.374251 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk4zs\" (UniqueName: \"kubernetes.io/projected/bf7b4339-b003-414f-a5d8-64c79be04e48-kube-api-access-xk4zs\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:38:25.899931 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:25.899902 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-pfqvt_bf7b4339-b003-414f-a5d8-64c79be04e48/registry-server/0.log" Jul 10 18:38:25.900375 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:25.899940 2576 generic.go:356] "Generic (PLEG): container finished" podID="bf7b4339-b003-414f-a5d8-64c79be04e48" containerID="cc52967b04da4e8aa832a2ea0c7161c7e21292a54a0b50f35e98bd8577c0972e" exitCode=2 Jul 10 18:38:25.900375 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:25.900021 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pfqvt" Jul 10 18:38:25.900375 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:25.900042 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pfqvt" event={"ID":"bf7b4339-b003-414f-a5d8-64c79be04e48","Type":"ContainerDied","Data":"cc52967b04da4e8aa832a2ea0c7161c7e21292a54a0b50f35e98bd8577c0972e"} Jul 10 18:38:25.900375 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:25.900092 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pfqvt" event={"ID":"bf7b4339-b003-414f-a5d8-64c79be04e48","Type":"ContainerDied","Data":"6ef926fd0dc57d9b0efe39a1c694264dea4dec8aa8671c22d99ee59bb1ee68e9"} Jul 10 18:38:25.900375 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:25.900113 2576 scope.go:117] "RemoveContainer" containerID="cc52967b04da4e8aa832a2ea0c7161c7e21292a54a0b50f35e98bd8577c0972e" Jul 10 18:38:25.908247 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:25.908230 2576 scope.go:117] "RemoveContainer" containerID="cc52967b04da4e8aa832a2ea0c7161c7e21292a54a0b50f35e98bd8577c0972e" Jul 10 18:38:25.908522 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:38:25.908476 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc52967b04da4e8aa832a2ea0c7161c7e21292a54a0b50f35e98bd8577c0972e\": container with ID starting with cc52967b04da4e8aa832a2ea0c7161c7e21292a54a0b50f35e98bd8577c0972e not found: ID does not exist" containerID="cc52967b04da4e8aa832a2ea0c7161c7e21292a54a0b50f35e98bd8577c0972e" Jul 10 18:38:25.908612 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:25.908528 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc52967b04da4e8aa832a2ea0c7161c7e21292a54a0b50f35e98bd8577c0972e"} err="failed to get container status \"cc52967b04da4e8aa832a2ea0c7161c7e21292a54a0b50f35e98bd8577c0972e\": rpc error: code = NotFound desc = could not find container \"cc52967b04da4e8aa832a2ea0c7161c7e21292a54a0b50f35e98bd8577c0972e\": container with ID starting with cc52967b04da4e8aa832a2ea0c7161c7e21292a54a0b50f35e98bd8577c0972e not found: ID does not exist" Jul 10 18:38:25.915567 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:25.915547 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pfqvt"] Jul 10 18:38:25.919611 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:25.919591 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pfqvt"] Jul 10 18:38:27.842424 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:27.842390 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf7b4339-b003-414f-a5d8-64c79be04e48" path="/var/lib/kubelet/pods/bf7b4339-b003-414f-a5d8-64c79be04e48/volumes" Jul 10 18:38:33.365314 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:33.365279 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-tbc99" Jul 10 18:38:33.365745 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:33.365334 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-tbc99" Jul 10 18:38:33.388066 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:33.388043 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-tbc99" Jul 10 18:38:33.957084 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:33.957057 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-tbc99" Jul 10 18:38:37.815887 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:37.815854 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6"] Jul 10 18:38:37.816255 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:37.816158 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf7b4339-b003-414f-a5d8-64c79be04e48" containerName="registry-server" Jul 10 18:38:37.816255 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:37.816169 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf7b4339-b003-414f-a5d8-64c79be04e48" containerName="registry-server" Jul 10 18:38:37.816255 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:37.816231 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf7b4339-b003-414f-a5d8-64c79be04e48" containerName="registry-server" Jul 10 18:38:37.818678 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:37.818661 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" Jul 10 18:38:37.820992 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:37.820970 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8jfkp\"" Jul 10 18:38:37.826916 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:37.826893 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6"] Jul 10 18:38:37.967894 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:37.967858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96002491-e88e-4839-9c4d-7801d0c622d4-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6\" (UID: \"96002491-e88e-4839-9c4d-7801d0c622d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" Jul 10 18:38:37.968044 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:37.967905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6f2f\" (UniqueName: \"kubernetes.io/projected/96002491-e88e-4839-9c4d-7801d0c622d4-kube-api-access-b6f2f\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6\" (UID: \"96002491-e88e-4839-9c4d-7801d0c622d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" Jul 10 18:38:37.968044 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:37.967983 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96002491-e88e-4839-9c4d-7801d0c622d4-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6\" (UID: \"96002491-e88e-4839-9c4d-7801d0c622d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" Jul 10 18:38:38.068467 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.068386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96002491-e88e-4839-9c4d-7801d0c622d4-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6\" (UID: \"96002491-e88e-4839-9c4d-7801d0c622d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" Jul 10 18:38:38.068467 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.068427 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6f2f\" (UniqueName: \"kubernetes.io/projected/96002491-e88e-4839-9c4d-7801d0c622d4-kube-api-access-b6f2f\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6\" (UID: \"96002491-e88e-4839-9c4d-7801d0c622d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" Jul 10 18:38:38.068467 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.068462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96002491-e88e-4839-9c4d-7801d0c622d4-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6\" (UID: \"96002491-e88e-4839-9c4d-7801d0c622d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" Jul 10 18:38:38.068770 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.068752 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96002491-e88e-4839-9c4d-7801d0c622d4-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6\" (UID: \"96002491-e88e-4839-9c4d-7801d0c622d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" Jul 10 18:38:38.068811 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.068776 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96002491-e88e-4839-9c4d-7801d0c622d4-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6\" (UID: \"96002491-e88e-4839-9c4d-7801d0c622d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" Jul 10 18:38:38.077052 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.077029 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6f2f\" (UniqueName: \"kubernetes.io/projected/96002491-e88e-4839-9c4d-7801d0c622d4-kube-api-access-b6f2f\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6\" (UID: \"96002491-e88e-4839-9c4d-7801d0c622d4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" Jul 10 18:38:38.127647 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.127613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" Jul 10 18:38:38.252688 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.252666 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6"] Jul 10 18:38:38.254647 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:38:38.254619 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96002491_e88e_4839_9c4d_7801d0c622d4.slice/crio-9a8cefd9be248a3513c98bb93424086fc28499f2a71cad29f758746e45260750 WatchSource:0}: Error finding container 9a8cefd9be248a3513c98bb93424086fc28499f2a71cad29f758746e45260750: Status 404 returned error can't find the container with id 9a8cefd9be248a3513c98bb93424086fc28499f2a71cad29f758746e45260750 Jul 10 18:38:38.446423 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.446392 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd"] Jul 10 18:38:38.448909 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.448893 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" Jul 10 18:38:38.457626 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.457602 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd"] Jul 10 18:38:38.470688 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.470663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/967cc74b-9c9f-4cd9-94aa-aa7199f74236-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd\" (UID: \"967cc74b-9c9f-4cd9-94aa-aa7199f74236\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" Jul 10 18:38:38.470811 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.470699 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/967cc74b-9c9f-4cd9-94aa-aa7199f74236-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd\" (UID: \"967cc74b-9c9f-4cd9-94aa-aa7199f74236\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" Jul 10 18:38:38.470811 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.470770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxcn5\" (UniqueName: \"kubernetes.io/projected/967cc74b-9c9f-4cd9-94aa-aa7199f74236-kube-api-access-xxcn5\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd\" (UID: \"967cc74b-9c9f-4cd9-94aa-aa7199f74236\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" Jul 10 18:38:38.571746 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.571712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxcn5\" (UniqueName: \"kubernetes.io/projected/967cc74b-9c9f-4cd9-94aa-aa7199f74236-kube-api-access-xxcn5\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd\" (UID: \"967cc74b-9c9f-4cd9-94aa-aa7199f74236\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" Jul 10 18:38:38.571928 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.571783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/967cc74b-9c9f-4cd9-94aa-aa7199f74236-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd\" (UID: \"967cc74b-9c9f-4cd9-94aa-aa7199f74236\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" Jul 10 18:38:38.571928 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.571822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/967cc74b-9c9f-4cd9-94aa-aa7199f74236-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd\" (UID: \"967cc74b-9c9f-4cd9-94aa-aa7199f74236\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" Jul 10 18:38:38.572181 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.572160 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/967cc74b-9c9f-4cd9-94aa-aa7199f74236-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd\" (UID: \"967cc74b-9c9f-4cd9-94aa-aa7199f74236\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" Jul 10 18:38:38.572243 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.572183 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/967cc74b-9c9f-4cd9-94aa-aa7199f74236-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd\" (UID: \"967cc74b-9c9f-4cd9-94aa-aa7199f74236\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" Jul 10 18:38:38.580097 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.580072 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxcn5\" (UniqueName: \"kubernetes.io/projected/967cc74b-9c9f-4cd9-94aa-aa7199f74236-kube-api-access-xxcn5\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd\" (UID: \"967cc74b-9c9f-4cd9-94aa-aa7199f74236\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" Jul 10 18:38:38.758308 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.758220 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" Jul 10 18:38:38.883620 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.883587 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd"] Jul 10 18:38:38.887736 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:38:38.887707 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967cc74b_9c9f_4cd9_94aa_aa7199f74236.slice/crio-fada93034b3b53c1bdbc3a5b350922e046d9cabda69135d7a86f7d66a42652c9 WatchSource:0}: Error finding container fada93034b3b53c1bdbc3a5b350922e046d9cabda69135d7a86f7d66a42652c9: Status 404 returned error can't find the container with id fada93034b3b53c1bdbc3a5b350922e046d9cabda69135d7a86f7d66a42652c9 Jul 10 18:38:38.958583 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.958545 2576 generic.go:356] "Generic (PLEG): container finished" podID="96002491-e88e-4839-9c4d-7801d0c622d4" containerID="5565fe944b95cc52f3635a32c85d5cf8e9d549d34ca0972f208d089d6844b452" exitCode=0 Jul 10 18:38:38.958699 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.958676 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" event={"ID":"96002491-e88e-4839-9c4d-7801d0c622d4","Type":"ContainerDied","Data":"5565fe944b95cc52f3635a32c85d5cf8e9d549d34ca0972f208d089d6844b452"} Jul 10 18:38:38.958767 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.958730 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" event={"ID":"96002491-e88e-4839-9c4d-7801d0c622d4","Type":"ContainerStarted","Data":"9a8cefd9be248a3513c98bb93424086fc28499f2a71cad29f758746e45260750"} Jul 10 18:38:38.960191 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.960163 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" event={"ID":"967cc74b-9c9f-4cd9-94aa-aa7199f74236","Type":"ContainerStarted","Data":"0944980a24aec4094c115976f94b54ea04cc52e1490da856763a08a7a5c8cc5e"} Jul 10 18:38:38.960337 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:38.960212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" event={"ID":"967cc74b-9c9f-4cd9-94aa-aa7199f74236","Type":"ContainerStarted","Data":"fada93034b3b53c1bdbc3a5b350922e046d9cabda69135d7a86f7d66a42652c9"} Jul 10 18:38:39.025806 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.025727 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr"] Jul 10 18:38:39.027987 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.027965 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" Jul 10 18:38:39.038090 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.037874 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr"] Jul 10 18:38:39.074811 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.074782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/720edcea-b029-45a0-b300-9eac320b1d73-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr\" (UID: \"720edcea-b029-45a0-b300-9eac320b1d73\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" Jul 10 18:38:39.074811 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.074815 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24ggz\" (UniqueName: \"kubernetes.io/projected/720edcea-b029-45a0-b300-9eac320b1d73-kube-api-access-24ggz\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr\" (UID: \"720edcea-b029-45a0-b300-9eac320b1d73\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" Jul 10 18:38:39.075006 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.074893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/720edcea-b029-45a0-b300-9eac320b1d73-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr\" (UID: \"720edcea-b029-45a0-b300-9eac320b1d73\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" Jul 10 18:38:39.175339 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.175306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/720edcea-b029-45a0-b300-9eac320b1d73-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr\" (UID: \"720edcea-b029-45a0-b300-9eac320b1d73\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" Jul 10 18:38:39.175339 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.175343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/720edcea-b029-45a0-b300-9eac320b1d73-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr\" (UID: \"720edcea-b029-45a0-b300-9eac320b1d73\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" Jul 10 18:38:39.175542 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.175385 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24ggz\" (UniqueName: \"kubernetes.io/projected/720edcea-b029-45a0-b300-9eac320b1d73-kube-api-access-24ggz\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr\" (UID: \"720edcea-b029-45a0-b300-9eac320b1d73\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" Jul 10 18:38:39.175677 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.175657 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/720edcea-b029-45a0-b300-9eac320b1d73-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr\" (UID: \"720edcea-b029-45a0-b300-9eac320b1d73\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" Jul 10 18:38:39.175756 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.175740 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/720edcea-b029-45a0-b300-9eac320b1d73-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr\" (UID: \"720edcea-b029-45a0-b300-9eac320b1d73\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" Jul 10 18:38:39.183156 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.183129 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24ggz\" (UniqueName: \"kubernetes.io/projected/720edcea-b029-45a0-b300-9eac320b1d73-kube-api-access-24ggz\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr\" (UID: \"720edcea-b029-45a0-b300-9eac320b1d73\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" Jul 10 18:38:39.340475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.340446 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" Jul 10 18:38:39.428637 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.428610 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz"] Jul 10 18:38:39.431094 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.431074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" Jul 10 18:38:39.440949 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.440856 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz"] Jul 10 18:38:39.471132 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.471075 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr"] Jul 10 18:38:39.473235 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:38:39.473209 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720edcea_b029_45a0_b300_9eac320b1d73.slice/crio-371a6c1e2bd406669b446672c51a76d94442fa98025a1bb9081d85b4c6969a3b WatchSource:0}: Error finding container 371a6c1e2bd406669b446672c51a76d94442fa98025a1bb9081d85b4c6969a3b: Status 404 returned error can't find the container with id 371a6c1e2bd406669b446672c51a76d94442fa98025a1bb9081d85b4c6969a3b Jul 10 18:38:39.478467 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.478438 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz\" (UID: \"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" Jul 10 18:38:39.478556 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.478527 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5t2\" (UniqueName: \"kubernetes.io/projected/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-kube-api-access-ww5t2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz\" (UID: \"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" Jul 10 18:38:39.478624 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.478582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz\" (UID: \"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" Jul 10 18:38:39.579005 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.578980 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz\" (UID: \"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" Jul 10 18:38:39.579111 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.579076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz\" (UID: \"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" Jul 10 18:38:39.579186 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.579113 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5t2\" (UniqueName: \"kubernetes.io/projected/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-kube-api-access-ww5t2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz\" (UID: \"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" Jul 10 18:38:39.579482 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.579463 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz\" (UID: \"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" Jul 10 18:38:39.579561 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.579459 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz\" (UID: \"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" Jul 10 18:38:39.587498 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.587473 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5t2\" (UniqueName: \"kubernetes.io/projected/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-kube-api-access-ww5t2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz\" (UID: \"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" Jul 10 18:38:39.743232 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.743156 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" Jul 10 18:38:39.876168 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.876143 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz"] Jul 10 18:38:39.877913 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:38:39.877888 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f1c0a6_2710_4d70_9ac0_e34c95f25bac.slice/crio-f73be135d5a55c6d105ad81decbe320ebc4c1ded897aacd1a8254bca36b651c9 WatchSource:0}: Error finding container f73be135d5a55c6d105ad81decbe320ebc4c1ded897aacd1a8254bca36b651c9: Status 404 returned error can't find the container with id f73be135d5a55c6d105ad81decbe320ebc4c1ded897aacd1a8254bca36b651c9 Jul 10 18:38:39.966496 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.966461 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" event={"ID":"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac","Type":"ContainerStarted","Data":"4fd72f3514b435bf6fc1806a0addc21d3c9e5808b4f6841a7496cd1faaddc256"} Jul 10 18:38:39.966910 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.966520 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" event={"ID":"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac","Type":"ContainerStarted","Data":"f73be135d5a55c6d105ad81decbe320ebc4c1ded897aacd1a8254bca36b651c9"} Jul 10 18:38:39.967944 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.967920 2576 generic.go:356] "Generic (PLEG): container finished" podID="720edcea-b029-45a0-b300-9eac320b1d73" containerID="dcfe01acef701fffa54c6d2d55a034d9d3590ad109f0d9cae779724855d96758" exitCode=0 Jul 10 18:38:39.968089 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.968001 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" event={"ID":"720edcea-b029-45a0-b300-9eac320b1d73","Type":"ContainerDied","Data":"dcfe01acef701fffa54c6d2d55a034d9d3590ad109f0d9cae779724855d96758"} Jul 10 18:38:39.968089 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.968057 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" event={"ID":"720edcea-b029-45a0-b300-9eac320b1d73","Type":"ContainerStarted","Data":"371a6c1e2bd406669b446672c51a76d94442fa98025a1bb9081d85b4c6969a3b"} Jul 10 18:38:39.969531 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.969505 2576 generic.go:356] "Generic (PLEG): container finished" podID="967cc74b-9c9f-4cd9-94aa-aa7199f74236" containerID="0944980a24aec4094c115976f94b54ea04cc52e1490da856763a08a7a5c8cc5e" exitCode=0 Jul 10 18:38:39.969766 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:39.969545 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" event={"ID":"967cc74b-9c9f-4cd9-94aa-aa7199f74236","Type":"ContainerDied","Data":"0944980a24aec4094c115976f94b54ea04cc52e1490da856763a08a7a5c8cc5e"} Jul 10 18:38:40.975212 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:40.975172 2576 generic.go:356] "Generic (PLEG): container finished" podID="b6f1c0a6-2710-4d70-9ac0-e34c95f25bac" containerID="4fd72f3514b435bf6fc1806a0addc21d3c9e5808b4f6841a7496cd1faaddc256" exitCode=0 Jul 10 18:38:40.975667 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:40.975256 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" event={"ID":"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac","Type":"ContainerDied","Data":"4fd72f3514b435bf6fc1806a0addc21d3c9e5808b4f6841a7496cd1faaddc256"} Jul 10 18:38:40.977098 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:40.977074 2576 generic.go:356] "Generic (PLEG): container finished" podID="96002491-e88e-4839-9c4d-7801d0c622d4" containerID="1dddf599727fa4b8891c49692af80b25e5e0e265df878315cbe30d0d38a25a52" exitCode=0 Jul 10 18:38:40.977220 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:40.977176 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" event={"ID":"96002491-e88e-4839-9c4d-7801d0c622d4","Type":"ContainerDied","Data":"1dddf599727fa4b8891c49692af80b25e5e0e265df878315cbe30d0d38a25a52"} Jul 10 18:38:40.979025 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:40.979000 2576 generic.go:356] "Generic (PLEG): container finished" podID="720edcea-b029-45a0-b300-9eac320b1d73" containerID="99f95a6df3984f0a694b21b01da65f27ecad12c1dc404749295a3ef9eaa3c5b9" exitCode=0 Jul 10 18:38:40.979120 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:40.979043 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" event={"ID":"720edcea-b029-45a0-b300-9eac320b1d73","Type":"ContainerDied","Data":"99f95a6df3984f0a694b21b01da65f27ecad12c1dc404749295a3ef9eaa3c5b9"} Jul 10 18:38:41.985290 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:41.985257 2576 generic.go:356] "Generic (PLEG): container finished" podID="96002491-e88e-4839-9c4d-7801d0c622d4" containerID="80a2fd7ab164b894e2fcc01e32465356517a6e666b62077926ba6cc89fdce9f9" exitCode=0 Jul 10 18:38:41.985839 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:41.985342 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" event={"ID":"96002491-e88e-4839-9c4d-7801d0c622d4","Type":"ContainerDied","Data":"80a2fd7ab164b894e2fcc01e32465356517a6e666b62077926ba6cc89fdce9f9"} Jul 10 18:38:41.987248 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:41.987226 2576 generic.go:356] "Generic (PLEG): container finished" podID="720edcea-b029-45a0-b300-9eac320b1d73" containerID="3424646f4e16fdff50079de7a0ff7318c0f054b80202c728be6694ad10bf91ba" exitCode=0 Jul 10 18:38:41.987374 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:41.987302 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" event={"ID":"720edcea-b029-45a0-b300-9eac320b1d73","Type":"ContainerDied","Data":"3424646f4e16fdff50079de7a0ff7318c0f054b80202c728be6694ad10bf91ba"} Jul 10 18:38:41.989026 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:41.989003 2576 generic.go:356] "Generic (PLEG): container finished" podID="967cc74b-9c9f-4cd9-94aa-aa7199f74236" containerID="b3b377e97641f7ebc9fa716d05001af7948ac0c8e14fcde4dcb0a936be63d63b" exitCode=0 Jul 10 18:38:41.989150 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:41.989079 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" event={"ID":"967cc74b-9c9f-4cd9-94aa-aa7199f74236","Type":"ContainerDied","Data":"b3b377e97641f7ebc9fa716d05001af7948ac0c8e14fcde4dcb0a936be63d63b"} Jul 10 18:38:41.990891 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:41.990870 2576 generic.go:356] "Generic (PLEG): container finished" podID="b6f1c0a6-2710-4d70-9ac0-e34c95f25bac" containerID="241742c44896ee4182b28fb65f2bf9f07e03bd9092f2c5f1265bc1723435b0e3" exitCode=0 Jul 10 18:38:41.990982 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:41.990913 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" event={"ID":"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac","Type":"ContainerDied","Data":"241742c44896ee4182b28fb65f2bf9f07e03bd9092f2c5f1265bc1723435b0e3"} Jul 10 18:38:42.998962 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:42.998931 2576 generic.go:356] "Generic (PLEG): container finished" podID="b6f1c0a6-2710-4d70-9ac0-e34c95f25bac" containerID="7972d01bb1d0e4afb5d61f174d0d2bbec333e9a6c98e90b4df3865f6985dffb1" exitCode=0 Jul 10 18:38:42.999335 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:42.998969 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" event={"ID":"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac","Type":"ContainerDied","Data":"7972d01bb1d0e4afb5d61f174d0d2bbec333e9a6c98e90b4df3865f6985dffb1"} Jul 10 18:38:43.000972 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.000949 2576 generic.go:356] "Generic (PLEG): container finished" podID="967cc74b-9c9f-4cd9-94aa-aa7199f74236" containerID="d51c8ff4ee310024abf9ef85d92f1ce99ded906dc9b7b707b45229a52ec74dff" exitCode=0 Jul 10 18:38:43.001103 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.001022 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" event={"ID":"967cc74b-9c9f-4cd9-94aa-aa7199f74236","Type":"ContainerDied","Data":"d51c8ff4ee310024abf9ef85d92f1ce99ded906dc9b7b707b45229a52ec74dff"} Jul 10 18:38:43.152970 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.152947 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" Jul 10 18:38:43.155705 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.155685 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" Jul 10 18:38:43.208768 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.208738 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24ggz\" (UniqueName: \"kubernetes.io/projected/720edcea-b029-45a0-b300-9eac320b1d73-kube-api-access-24ggz\") pod \"720edcea-b029-45a0-b300-9eac320b1d73\" (UID: \"720edcea-b029-45a0-b300-9eac320b1d73\") " Jul 10 18:38:43.208768 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.208770 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/720edcea-b029-45a0-b300-9eac320b1d73-util\") pod \"720edcea-b029-45a0-b300-9eac320b1d73\" (UID: \"720edcea-b029-45a0-b300-9eac320b1d73\") " Jul 10 18:38:43.208952 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.208797 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96002491-e88e-4839-9c4d-7801d0c622d4-util\") pod \"96002491-e88e-4839-9c4d-7801d0c622d4\" (UID: \"96002491-e88e-4839-9c4d-7801d0c622d4\") " Jul 10 18:38:43.208952 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.208831 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96002491-e88e-4839-9c4d-7801d0c622d4-bundle\") pod \"96002491-e88e-4839-9c4d-7801d0c622d4\" (UID: \"96002491-e88e-4839-9c4d-7801d0c622d4\") " Jul 10 18:38:43.208952 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.208857 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6f2f\" (UniqueName: \"kubernetes.io/projected/96002491-e88e-4839-9c4d-7801d0c622d4-kube-api-access-b6f2f\") pod \"96002491-e88e-4839-9c4d-7801d0c622d4\" (UID: \"96002491-e88e-4839-9c4d-7801d0c622d4\") " Jul 10 18:38:43.208952 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.208883 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/720edcea-b029-45a0-b300-9eac320b1d73-bundle\") pod \"720edcea-b029-45a0-b300-9eac320b1d73\" (UID: \"720edcea-b029-45a0-b300-9eac320b1d73\") " Jul 10 18:38:43.209436 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.209408 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96002491-e88e-4839-9c4d-7801d0c622d4-bundle" (OuterVolumeSpecName: "bundle") pod "96002491-e88e-4839-9c4d-7801d0c622d4" (UID: "96002491-e88e-4839-9c4d-7801d0c622d4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:43.209563 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.209523 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720edcea-b029-45a0-b300-9eac320b1d73-bundle" (OuterVolumeSpecName: "bundle") pod "720edcea-b029-45a0-b300-9eac320b1d73" (UID: "720edcea-b029-45a0-b300-9eac320b1d73"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:43.210997 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.210978 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96002491-e88e-4839-9c4d-7801d0c622d4-kube-api-access-b6f2f" (OuterVolumeSpecName: "kube-api-access-b6f2f") pod "96002491-e88e-4839-9c4d-7801d0c622d4" (UID: "96002491-e88e-4839-9c4d-7801d0c622d4"). InnerVolumeSpecName "kube-api-access-b6f2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:38:43.211265 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.211235 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720edcea-b029-45a0-b300-9eac320b1d73-kube-api-access-24ggz" (OuterVolumeSpecName: "kube-api-access-24ggz") pod "720edcea-b029-45a0-b300-9eac320b1d73" (UID: "720edcea-b029-45a0-b300-9eac320b1d73"). InnerVolumeSpecName "kube-api-access-24ggz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:38:43.214848 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.214827 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96002491-e88e-4839-9c4d-7801d0c622d4-util" (OuterVolumeSpecName: "util") pod "96002491-e88e-4839-9c4d-7801d0c622d4" (UID: "96002491-e88e-4839-9c4d-7801d0c622d4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:43.215500 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.215478 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720edcea-b029-45a0-b300-9eac320b1d73-util" (OuterVolumeSpecName: "util") pod "720edcea-b029-45a0-b300-9eac320b1d73" (UID: "720edcea-b029-45a0-b300-9eac320b1d73"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:43.310455 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.310377 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96002491-e88e-4839-9c4d-7801d0c622d4-util\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:38:43.310455 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.310402 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96002491-e88e-4839-9c4d-7801d0c622d4-bundle\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:38:43.310455 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.310411 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6f2f\" (UniqueName: \"kubernetes.io/projected/96002491-e88e-4839-9c4d-7801d0c622d4-kube-api-access-b6f2f\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:38:43.310455 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.310420 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/720edcea-b029-45a0-b300-9eac320b1d73-bundle\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:38:43.310455 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.310429 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24ggz\" (UniqueName: \"kubernetes.io/projected/720edcea-b029-45a0-b300-9eac320b1d73-kube-api-access-24ggz\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:38:43.310455 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:43.310436 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/720edcea-b029-45a0-b300-9eac320b1d73-util\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:38:44.007132 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.007100 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" Jul 10 18:38:44.007132 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.007111 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6" event={"ID":"96002491-e88e-4839-9c4d-7801d0c622d4","Type":"ContainerDied","Data":"9a8cefd9be248a3513c98bb93424086fc28499f2a71cad29f758746e45260750"} Jul 10 18:38:44.007618 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.007161 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a8cefd9be248a3513c98bb93424086fc28499f2a71cad29f758746e45260750" Jul 10 18:38:44.009130 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.009080 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" Jul 10 18:38:44.009130 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.009082 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr" event={"ID":"720edcea-b029-45a0-b300-9eac320b1d73","Type":"ContainerDied","Data":"371a6c1e2bd406669b446672c51a76d94442fa98025a1bb9081d85b4c6969a3b"} Jul 10 18:38:44.009305 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.009131 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="371a6c1e2bd406669b446672c51a76d94442fa98025a1bb9081d85b4c6969a3b" Jul 10 18:38:44.148533 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.148511 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" Jul 10 18:38:44.166836 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.166817 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" Jul 10 18:38:44.218626 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.218594 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/967cc74b-9c9f-4cd9-94aa-aa7199f74236-bundle\") pod \"967cc74b-9c9f-4cd9-94aa-aa7199f74236\" (UID: \"967cc74b-9c9f-4cd9-94aa-aa7199f74236\") " Jul 10 18:38:44.218626 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.218631 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/967cc74b-9c9f-4cd9-94aa-aa7199f74236-util\") pod \"967cc74b-9c9f-4cd9-94aa-aa7199f74236\" (UID: \"967cc74b-9c9f-4cd9-94aa-aa7199f74236\") " Jul 10 18:38:44.218844 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.218679 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-bundle\") pod \"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac\" (UID: \"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac\") " Jul 10 18:38:44.218844 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.218739 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww5t2\" (UniqueName: \"kubernetes.io/projected/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-kube-api-access-ww5t2\") pod \"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac\" (UID: \"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac\") " Jul 10 18:38:44.218844 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.218765 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxcn5\" (UniqueName: \"kubernetes.io/projected/967cc74b-9c9f-4cd9-94aa-aa7199f74236-kube-api-access-xxcn5\") pod \"967cc74b-9c9f-4cd9-94aa-aa7199f74236\" (UID: \"967cc74b-9c9f-4cd9-94aa-aa7199f74236\") " Jul 10 18:38:44.218844 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.218792 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-util\") pod \"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac\" (UID: \"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac\") " Jul 10 18:38:44.219238 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.219206 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967cc74b-9c9f-4cd9-94aa-aa7199f74236-bundle" (OuterVolumeSpecName: "bundle") pod "967cc74b-9c9f-4cd9-94aa-aa7199f74236" (UID: "967cc74b-9c9f-4cd9-94aa-aa7199f74236"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:44.219463 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.219424 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-bundle" (OuterVolumeSpecName: "bundle") pod "b6f1c0a6-2710-4d70-9ac0-e34c95f25bac" (UID: "b6f1c0a6-2710-4d70-9ac0-e34c95f25bac"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:44.220808 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.220786 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-kube-api-access-ww5t2" (OuterVolumeSpecName: "kube-api-access-ww5t2") pod "b6f1c0a6-2710-4d70-9ac0-e34c95f25bac" (UID: "b6f1c0a6-2710-4d70-9ac0-e34c95f25bac"). InnerVolumeSpecName "kube-api-access-ww5t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:38:44.220904 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.220868 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967cc74b-9c9f-4cd9-94aa-aa7199f74236-kube-api-access-xxcn5" (OuterVolumeSpecName: "kube-api-access-xxcn5") pod "967cc74b-9c9f-4cd9-94aa-aa7199f74236" (UID: "967cc74b-9c9f-4cd9-94aa-aa7199f74236"). InnerVolumeSpecName "kube-api-access-xxcn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:38:44.224751 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.224723 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-util" (OuterVolumeSpecName: "util") pod "b6f1c0a6-2710-4d70-9ac0-e34c95f25bac" (UID: "b6f1c0a6-2710-4d70-9ac0-e34c95f25bac"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:44.299673 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.299638 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967cc74b-9c9f-4cd9-94aa-aa7199f74236-util" (OuterVolumeSpecName: "util") pod "967cc74b-9c9f-4cd9-94aa-aa7199f74236" (UID: "967cc74b-9c9f-4cd9-94aa-aa7199f74236"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:38:44.319737 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.319706 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww5t2\" (UniqueName: \"kubernetes.io/projected/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-kube-api-access-ww5t2\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:38:44.319737 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.319735 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxcn5\" (UniqueName: \"kubernetes.io/projected/967cc74b-9c9f-4cd9-94aa-aa7199f74236-kube-api-access-xxcn5\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:38:44.319883 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.319745 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-util\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:38:44.319883 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.319757 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/967cc74b-9c9f-4cd9-94aa-aa7199f74236-bundle\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:38:44.319883 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.319769 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/967cc74b-9c9f-4cd9-94aa-aa7199f74236-util\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:38:44.319883 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:44.319778 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6f1c0a6-2710-4d70-9ac0-e34c95f25bac-bundle\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:38:45.016406 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:45.016381 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" Jul 10 18:38:45.016804 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:45.016380 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd" event={"ID":"967cc74b-9c9f-4cd9-94aa-aa7199f74236","Type":"ContainerDied","Data":"fada93034b3b53c1bdbc3a5b350922e046d9cabda69135d7a86f7d66a42652c9"} Jul 10 18:38:45.016804 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:45.016504 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fada93034b3b53c1bdbc3a5b350922e046d9cabda69135d7a86f7d66a42652c9" Jul 10 18:38:45.018176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:45.018151 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" event={"ID":"b6f1c0a6-2710-4d70-9ac0-e34c95f25bac","Type":"ContainerDied","Data":"f73be135d5a55c6d105ad81decbe320ebc4c1ded897aacd1a8254bca36b651c9"} Jul 10 18:38:45.018271 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:45.018183 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz" Jul 10 18:38:45.018271 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:45.018187 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f73be135d5a55c6d105ad81decbe320ebc4c1ded897aacd1a8254bca36b651c9" Jul 10 18:38:55.253053 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253019 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-nmtvc"] Jul 10 18:38:55.253477 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253374 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f1c0a6-2710-4d70-9ac0-e34c95f25bac" containerName="extract" Jul 10 18:38:55.253477 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253392 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f1c0a6-2710-4d70-9ac0-e34c95f25bac" containerName="extract" Jul 10 18:38:55.253477 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253408 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96002491-e88e-4839-9c4d-7801d0c622d4" containerName="util" Jul 10 18:38:55.253477 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253416 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="96002491-e88e-4839-9c4d-7801d0c622d4" containerName="util" Jul 10 18:38:55.253477 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253426 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96002491-e88e-4839-9c4d-7801d0c622d4" containerName="pull" Jul 10 18:38:55.253477 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253432 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="96002491-e88e-4839-9c4d-7801d0c622d4" containerName="pull" Jul 10 18:38:55.253477 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253441 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="720edcea-b029-45a0-b300-9eac320b1d73" containerName="pull" Jul 10 18:38:55.253477 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253446 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="720edcea-b029-45a0-b300-9eac320b1d73" containerName="pull" Jul 10 18:38:55.253477 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253452 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967cc74b-9c9f-4cd9-94aa-aa7199f74236" containerName="pull" Jul 10 18:38:55.253477 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253457 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="967cc74b-9c9f-4cd9-94aa-aa7199f74236" containerName="pull" Jul 10 18:38:55.253477 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253466 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96002491-e88e-4839-9c4d-7801d0c622d4" containerName="extract" Jul 10 18:38:55.253477 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253472 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="96002491-e88e-4839-9c4d-7801d0c622d4" containerName="extract" Jul 10 18:38:55.253477 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253479 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967cc74b-9c9f-4cd9-94aa-aa7199f74236" containerName="util" Jul 10 18:38:55.253477 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253484 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="967cc74b-9c9f-4cd9-94aa-aa7199f74236" containerName="util" Jul 10 18:38:55.253477 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253489 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f1c0a6-2710-4d70-9ac0-e34c95f25bac" containerName="util" Jul 10 18:38:55.254032 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253496 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f1c0a6-2710-4d70-9ac0-e34c95f25bac" containerName="util" Jul 10 18:38:55.254032 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253507 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="720edcea-b029-45a0-b300-9eac320b1d73" containerName="extract" Jul 10 18:38:55.254032 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253515 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="720edcea-b029-45a0-b300-9eac320b1d73" containerName="extract" Jul 10 18:38:55.254032 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253531 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="720edcea-b029-45a0-b300-9eac320b1d73" containerName="util" Jul 10 18:38:55.254032 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253539 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="720edcea-b029-45a0-b300-9eac320b1d73" containerName="util" Jul 10 18:38:55.254032 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253544 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967cc74b-9c9f-4cd9-94aa-aa7199f74236" containerName="extract" Jul 10 18:38:55.254032 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253549 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="967cc74b-9c9f-4cd9-94aa-aa7199f74236" containerName="extract" Jul 10 18:38:55.254032 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253560 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f1c0a6-2710-4d70-9ac0-e34c95f25bac" containerName="pull" Jul 10 18:38:55.254032 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253565 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f1c0a6-2710-4d70-9ac0-e34c95f25bac" containerName="pull" Jul 10 18:38:55.254032 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253621 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="967cc74b-9c9f-4cd9-94aa-aa7199f74236" containerName="extract" Jul 10 18:38:55.254032 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253628 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f1c0a6-2710-4d70-9ac0-e34c95f25bac" containerName="extract" Jul 10 18:38:55.254032 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253635 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="96002491-e88e-4839-9c4d-7801d0c622d4" containerName="extract" Jul 10 18:38:55.254032 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.253642 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="720edcea-b029-45a0-b300-9eac320b1d73" containerName="extract" Jul 10 18:38:55.260509 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.260492 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nmtvc" Jul 10 18:38:55.266290 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.266261 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2zmhb\"" Jul 10 18:38:55.269637 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.269614 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-nmtvc"] Jul 10 18:38:55.304132 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.304098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45sfj\" (UniqueName: \"kubernetes.io/projected/06cfb935-4eed-451d-a62e-c44fb9174a7c-kube-api-access-45sfj\") pod \"authorino-operator-75cc8d5cb7-nmtvc\" (UID: \"06cfb935-4eed-451d-a62e-c44fb9174a7c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-nmtvc" Jul 10 18:38:55.405428 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.405391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45sfj\" (UniqueName: \"kubernetes.io/projected/06cfb935-4eed-451d-a62e-c44fb9174a7c-kube-api-access-45sfj\") pod \"authorino-operator-75cc8d5cb7-nmtvc\" (UID: \"06cfb935-4eed-451d-a62e-c44fb9174a7c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-nmtvc" Jul 10 18:38:55.413667 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.413637 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45sfj\" (UniqueName: \"kubernetes.io/projected/06cfb935-4eed-451d-a62e-c44fb9174a7c-kube-api-access-45sfj\") pod \"authorino-operator-75cc8d5cb7-nmtvc\" (UID: \"06cfb935-4eed-451d-a62e-c44fb9174a7c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-nmtvc" Jul 10 18:38:55.572185 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.572160 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nmtvc" Jul 10 18:38:55.698844 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:55.698814 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-nmtvc"] Jul 10 18:38:55.702177 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:38:55.702151 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cfb935_4eed_451d_a62e_c44fb9174a7c.slice/crio-9ea4c31831fe863e22ebe87eddb58be644c5686216a3106fa3123ee724548be3 WatchSource:0}: Error finding container 9ea4c31831fe863e22ebe87eddb58be644c5686216a3106fa3123ee724548be3: Status 404 returned error can't find the container with id 9ea4c31831fe863e22ebe87eddb58be644c5686216a3106fa3123ee724548be3 Jul 10 18:38:56.069968 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:56.069939 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nmtvc" event={"ID":"06cfb935-4eed-451d-a62e-c44fb9174a7c","Type":"ContainerStarted","Data":"9ea4c31831fe863e22ebe87eddb58be644c5686216a3106fa3123ee724548be3"} Jul 10 18:38:58.516847 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:58.516765 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt"] Jul 10 18:38:58.520056 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:58.520036 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" Jul 10 18:38:58.522384 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:58.522370 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-t69vq\"" Jul 10 18:38:58.530825 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:58.530804 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt"] Jul 10 18:38:58.631721 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:58.631690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcpng\" (UniqueName: \"kubernetes.io/projected/130ce75b-083d-4425-9407-e0b93e05c8f8-kube-api-access-fcpng\") pod \"limitador-operator-controller-manager-859d7b78bd-dw8rt\" (UID: \"130ce75b-083d-4425-9407-e0b93e05c8f8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" Jul 10 18:38:58.732931 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:58.732900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpng\" (UniqueName: \"kubernetes.io/projected/130ce75b-083d-4425-9407-e0b93e05c8f8-kube-api-access-fcpng\") pod \"limitador-operator-controller-manager-859d7b78bd-dw8rt\" (UID: \"130ce75b-083d-4425-9407-e0b93e05c8f8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" Jul 10 18:38:58.744796 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:58.744766 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcpng\" (UniqueName: \"kubernetes.io/projected/130ce75b-083d-4425-9407-e0b93e05c8f8-kube-api-access-fcpng\") pod \"limitador-operator-controller-manager-859d7b78bd-dw8rt\" (UID: \"130ce75b-083d-4425-9407-e0b93e05c8f8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" Jul 10 18:38:58.830278 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:58.830246 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" Jul 10 18:38:59.011952 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:59.011929 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt"] Jul 10 18:38:59.013524 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:38:59.013497 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130ce75b_083d_4425_9407_e0b93e05c8f8.slice/crio-f74edaf44a2cd2887269e8a0c3f31ccfd9e6298156b6ad380e67f12349111b50 WatchSource:0}: Error finding container f74edaf44a2cd2887269e8a0c3f31ccfd9e6298156b6ad380e67f12349111b50: Status 404 returned error can't find the container with id f74edaf44a2cd2887269e8a0c3f31ccfd9e6298156b6ad380e67f12349111b50 Jul 10 18:38:59.086670 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:59.086590 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nmtvc" event={"ID":"06cfb935-4eed-451d-a62e-c44fb9174a7c","Type":"ContainerStarted","Data":"ecdadd4171162e22fe30f091ce4baeeaae4592f01f9008a02698f4b94fe3a165"} Jul 10 18:38:59.087116 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:59.087041 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nmtvc" Jul 10 18:38:59.093567 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:59.093541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" event={"ID":"130ce75b-083d-4425-9407-e0b93e05c8f8","Type":"ContainerStarted","Data":"f74edaf44a2cd2887269e8a0c3f31ccfd9e6298156b6ad380e67f12349111b50"} Jul 10 18:38:59.117216 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:38:59.117171 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nmtvc" podStartSLOduration=1.573724543 podStartE2EDuration="4.117158943s" totalImagesPullingTime="2.5434344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:38:59.114861559 +0000 UTC m=+585.792574022" watchObservedRunningTime="2026-07-10 18:38:59.117158943 +0000 UTC m=+585.794871402" Jul 10 18:39:02.108627 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:02.108593 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" event={"ID":"130ce75b-083d-4425-9407-e0b93e05c8f8","Type":"ContainerStarted","Data":"c9f3473874b1273398e39270d93674df76635c10b391568eb3a0b72b9fbe838b"} Jul 10 18:39:02.109067 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:02.108684 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" Jul 10 18:39:02.125837 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:02.125782 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" podStartSLOduration=1.636073352 podStartE2EDuration="4.125765686s" totalImagesPullingTime="2.489692334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:02.12362044 +0000 UTC m=+588.801332894" watchObservedRunningTime="2026-07-10 18:39:02.125765686 +0000 UTC m=+588.803478148" Jul 10 18:39:06.668397 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:06.668345 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-2bpss"] Jul 10 18:39:06.671617 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:06.671601 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2bpss" Jul 10 18:39:06.674065 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:06.674035 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 18:39:06.674188 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:06.674085 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-j9ffm\"" Jul 10 18:39:06.684054 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:06.684028 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-2bpss"] Jul 10 18:39:06.803683 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:06.803639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvgv\" (UniqueName: \"kubernetes.io/projected/d3170799-2ee0-4649-b100-efef023e67e6-kube-api-access-nfvgv\") pod \"dns-operator-controller-manager-864dd78c4c-2bpss\" (UID: \"d3170799-2ee0-4649-b100-efef023e67e6\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2bpss" Jul 10 18:39:06.905078 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:06.905033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvgv\" (UniqueName: \"kubernetes.io/projected/d3170799-2ee0-4649-b100-efef023e67e6-kube-api-access-nfvgv\") pod \"dns-operator-controller-manager-864dd78c4c-2bpss\" (UID: \"d3170799-2ee0-4649-b100-efef023e67e6\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2bpss" Jul 10 18:39:06.913198 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:06.913161 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvgv\" (UniqueName: \"kubernetes.io/projected/d3170799-2ee0-4649-b100-efef023e67e6-kube-api-access-nfvgv\") pod \"dns-operator-controller-manager-864dd78c4c-2bpss\" (UID: \"d3170799-2ee0-4649-b100-efef023e67e6\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2bpss" Jul 10 18:39:06.984309 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:06.984222 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2bpss" Jul 10 18:39:07.112046 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:07.112023 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-2bpss"] Jul 10 18:39:07.114319 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:39:07.114291 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3170799_2ee0_4649_b100_efef023e67e6.slice/crio-20cb3851f2990285c9a52e962366cc876b15749423ed96693545bdd15df12d75 WatchSource:0}: Error finding container 20cb3851f2990285c9a52e962366cc876b15749423ed96693545bdd15df12d75: Status 404 returned error can't find the container with id 20cb3851f2990285c9a52e962366cc876b15749423ed96693545bdd15df12d75 Jul 10 18:39:07.134260 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:07.134230 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2bpss" event={"ID":"d3170799-2ee0-4649-b100-efef023e67e6","Type":"ContainerStarted","Data":"20cb3851f2990285c9a52e962366cc876b15749423ed96693545bdd15df12d75"} Jul 10 18:39:10.100942 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:10.100914 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nmtvc" Jul 10 18:39:10.153742 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:10.153706 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2bpss" event={"ID":"d3170799-2ee0-4649-b100-efef023e67e6","Type":"ContainerStarted","Data":"06c4894b741ad36be05fb2e8cfc65b240ea790b605af70c884c610df9d92e2b8"} Jul 10 18:39:10.153900 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:10.153799 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2bpss" Jul 10 18:39:10.174247 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:10.174201 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2bpss" podStartSLOduration=2.140681526 podStartE2EDuration="4.174186163s" totalImagesPullingTime="2.033504637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:10.172165123 +0000 UTC m=+596.849877585" watchObservedRunningTime="2026-07-10 18:39:10.174186163 +0000 UTC m=+596.851898624" Jul 10 18:39:12.661834 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.661800 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r"] Jul 10 18:39:12.666464 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.666443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r" Jul 10 18:39:12.668827 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.668802 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 18:39:12.668827 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.668814 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 18:39:12.668827 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.668805 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8jfkp\"" Jul 10 18:39:12.673020 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.672998 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r"] Jul 10 18:39:12.755159 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.755123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe032d86-7163-4e5b-9403-0b1804b79c1a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6zw5r\" (UID: \"fe032d86-7163-4e5b-9403-0b1804b79c1a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r" Jul 10 18:39:12.755314 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.755214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjwxx\" (UniqueName: \"kubernetes.io/projected/fe032d86-7163-4e5b-9403-0b1804b79c1a-kube-api-access-jjwxx\") pod \"kuadrant-console-plugin-66b5774985-6zw5r\" (UID: \"fe032d86-7163-4e5b-9403-0b1804b79c1a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r" Jul 10 18:39:12.755314 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.755245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fe032d86-7163-4e5b-9403-0b1804b79c1a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-6zw5r\" (UID: \"fe032d86-7163-4e5b-9403-0b1804b79c1a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r" Jul 10 18:39:12.856004 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.855966 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe032d86-7163-4e5b-9403-0b1804b79c1a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6zw5r\" (UID: \"fe032d86-7163-4e5b-9403-0b1804b79c1a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r" Jul 10 18:39:12.856174 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.856029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjwxx\" (UniqueName: \"kubernetes.io/projected/fe032d86-7163-4e5b-9403-0b1804b79c1a-kube-api-access-jjwxx\") pod \"kuadrant-console-plugin-66b5774985-6zw5r\" (UID: \"fe032d86-7163-4e5b-9403-0b1804b79c1a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r" Jul 10 18:39:12.856174 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.856054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fe032d86-7163-4e5b-9403-0b1804b79c1a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-6zw5r\" (UID: \"fe032d86-7163-4e5b-9403-0b1804b79c1a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r" Jul 10 18:39:12.856752 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.856732 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fe032d86-7163-4e5b-9403-0b1804b79c1a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-6zw5r\" (UID: \"fe032d86-7163-4e5b-9403-0b1804b79c1a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r" Jul 10 18:39:12.858407 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.858386 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe032d86-7163-4e5b-9403-0b1804b79c1a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6zw5r\" (UID: \"fe032d86-7163-4e5b-9403-0b1804b79c1a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r" Jul 10 18:39:12.863905 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.863878 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjwxx\" (UniqueName: \"kubernetes.io/projected/fe032d86-7163-4e5b-9403-0b1804b79c1a-kube-api-access-jjwxx\") pod \"kuadrant-console-plugin-66b5774985-6zw5r\" (UID: \"fe032d86-7163-4e5b-9403-0b1804b79c1a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r" Jul 10 18:39:12.976212 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:12.976129 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r" Jul 10 18:39:13.103800 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:13.103775 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r"] Jul 10 18:39:13.105565 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:39:13.105529 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe032d86_7163_4e5b_9403_0b1804b79c1a.slice/crio-4f6021b9cf52a0fd85108533170167e70675b5cf53dad09e1d25e0a9b28a9a92 WatchSource:0}: Error finding container 4f6021b9cf52a0fd85108533170167e70675b5cf53dad09e1d25e0a9b28a9a92: Status 404 returned error can't find the container with id 4f6021b9cf52a0fd85108533170167e70675b5cf53dad09e1d25e0a9b28a9a92 Jul 10 18:39:13.116437 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:13.116416 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" Jul 10 18:39:13.168906 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:13.168872 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r" event={"ID":"fe032d86-7163-4e5b-9403-0b1804b79c1a","Type":"ContainerStarted","Data":"4f6021b9cf52a0fd85108533170167e70675b5cf53dad09e1d25e0a9b28a9a92"} Jul 10 18:39:13.774069 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:13.774040 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:39:13.777687 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:13.777665 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:39:21.160593 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:21.160561 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2bpss" Jul 10 18:39:22.717910 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:22.717876 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4"] Jul 10 18:39:22.726248 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:22.726229 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4" Jul 10 18:39:22.728945 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:22.728922 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x9lc4\"" Jul 10 18:39:22.734492 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:22.734428 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4"] Jul 10 18:39:22.842743 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:22.842701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzpd4\" (UniqueName: \"kubernetes.io/projected/68a73088-fb31-4934-a10c-092d42453ff5-kube-api-access-mzpd4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kjpv4\" (UID: \"68a73088-fb31-4934-a10c-092d42453ff5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4" Jul 10 18:39:22.842743 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:22.842748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68a73088-fb31-4934-a10c-092d42453ff5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kjpv4\" (UID: \"68a73088-fb31-4934-a10c-092d42453ff5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4" Jul 10 18:39:22.944083 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:22.944039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzpd4\" (UniqueName: \"kubernetes.io/projected/68a73088-fb31-4934-a10c-092d42453ff5-kube-api-access-mzpd4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kjpv4\" (UID: \"68a73088-fb31-4934-a10c-092d42453ff5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4" Jul 10 18:39:22.944083 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:22.944086 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68a73088-fb31-4934-a10c-092d42453ff5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kjpv4\" (UID: \"68a73088-fb31-4934-a10c-092d42453ff5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4" Jul 10 18:39:22.944584 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:22.944561 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68a73088-fb31-4934-a10c-092d42453ff5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kjpv4\" (UID: \"68a73088-fb31-4934-a10c-092d42453ff5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4" Jul 10 18:39:22.952548 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:22.952512 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzpd4\" (UniqueName: \"kubernetes.io/projected/68a73088-fb31-4934-a10c-092d42453ff5-kube-api-access-mzpd4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kjpv4\" (UID: \"68a73088-fb31-4934-a10c-092d42453ff5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4" Jul 10 18:39:23.039457 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.039338 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4" Jul 10 18:39:23.397304 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.397273 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4"] Jul 10 18:39:23.407016 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.406986 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4"] Jul 10 18:39:23.419330 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.419306 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2"] Jul 10 18:39:23.424011 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.423986 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" Jul 10 18:39:23.436919 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.436896 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2"] Jul 10 18:39:23.442227 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.442197 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt"] Jul 10 18:39:23.442499 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.442451 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" podUID="130ce75b-083d-4425-9407-e0b93e05c8f8" containerName="manager" containerID="cri-o://c9f3473874b1273398e39270d93674df76635c10b391568eb3a0b72b9fbe838b" gracePeriod=2 Jul 10 18:39:23.463915 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.463881 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt"] Jul 10 18:39:23.471804 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:39:23.471765 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dw8rt\" is forbidden: User \"system:node:ip-10-0-140-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-180.ec2.internal' and this object" podUID="130ce75b-083d-4425-9407-e0b93e05c8f8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" Jul 10 18:39:23.549939 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.549900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv6km\" (UniqueName: \"kubernetes.io/projected/f471da5e-2233-4cee-83d8-b17b52006e70-kube-api-access-tv6km\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zhqt2\" (UID: \"f471da5e-2233-4cee-83d8-b17b52006e70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" Jul 10 18:39:23.550115 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.549987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f471da5e-2233-4cee-83d8-b17b52006e70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zhqt2\" (UID: \"f471da5e-2233-4cee-83d8-b17b52006e70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" Jul 10 18:39:23.650949 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.650867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv6km\" (UniqueName: \"kubernetes.io/projected/f471da5e-2233-4cee-83d8-b17b52006e70-kube-api-access-tv6km\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zhqt2\" (UID: \"f471da5e-2233-4cee-83d8-b17b52006e70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" Jul 10 18:39:23.650949 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.650931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f471da5e-2233-4cee-83d8-b17b52006e70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zhqt2\" (UID: \"f471da5e-2233-4cee-83d8-b17b52006e70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" Jul 10 18:39:23.651338 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.651314 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f471da5e-2233-4cee-83d8-b17b52006e70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zhqt2\" (UID: \"f471da5e-2233-4cee-83d8-b17b52006e70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" Jul 10 18:39:23.662889 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.662860 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv6km\" (UniqueName: \"kubernetes.io/projected/f471da5e-2233-4cee-83d8-b17b52006e70-kube-api-access-tv6km\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zhqt2\" (UID: \"f471da5e-2233-4cee-83d8-b17b52006e70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" Jul 10 18:39:23.738814 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:23.738772 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" Jul 10 18:39:23.843102 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:39:23.843060 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dw8rt\" is forbidden: User \"system:node:ip-10-0-140-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-180.ec2.internal' and this object" podUID="130ce75b-083d-4425-9407-e0b93e05c8f8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" Jul 10 18:39:36.118651 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:36.118628 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2"] Jul 10 18:39:36.158320 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:39:36.158283 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf471da5e_2233_4cee_83d8_b17b52006e70.slice/crio-ad85af2f82bd69ee222ac498e53bbf371f1ccb6886bced21792eb20df1e14b93 WatchSource:0}: Error finding container ad85af2f82bd69ee222ac498e53bbf371f1ccb6886bced21792eb20df1e14b93: Status 404 returned error can't find the container with id ad85af2f82bd69ee222ac498e53bbf371f1ccb6886bced21792eb20df1e14b93 Jul 10 18:39:36.186258 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:36.186235 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" Jul 10 18:39:36.188460 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:39:36.188434 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dw8rt\" is forbidden: User \"system:node:ip-10-0-140-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-180.ec2.internal' and this object" podUID="130ce75b-083d-4425-9407-e0b93e05c8f8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" Jul 10 18:39:36.254404 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:36.254326 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcpng\" (UniqueName: \"kubernetes.io/projected/130ce75b-083d-4425-9407-e0b93e05c8f8-kube-api-access-fcpng\") pod \"130ce75b-083d-4425-9407-e0b93e05c8f8\" (UID: \"130ce75b-083d-4425-9407-e0b93e05c8f8\") " Jul 10 18:39:36.256157 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:36.256130 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130ce75b-083d-4425-9407-e0b93e05c8f8-kube-api-access-fcpng" (OuterVolumeSpecName: "kube-api-access-fcpng") pod "130ce75b-083d-4425-9407-e0b93e05c8f8" (UID: "130ce75b-083d-4425-9407-e0b93e05c8f8"). InnerVolumeSpecName "kube-api-access-fcpng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:39:36.296838 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:36.296806 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" event={"ID":"f471da5e-2233-4cee-83d8-b17b52006e70","Type":"ContainerStarted","Data":"ad85af2f82bd69ee222ac498e53bbf371f1ccb6886bced21792eb20df1e14b93"} Jul 10 18:39:36.297953 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:36.297924 2576 generic.go:356] "Generic (PLEG): container finished" podID="130ce75b-083d-4425-9407-e0b93e05c8f8" containerID="c9f3473874b1273398e39270d93674df76635c10b391568eb3a0b72b9fbe838b" exitCode=0 Jul 10 18:39:36.298074 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:36.297976 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" Jul 10 18:39:36.298074 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:36.298024 2576 scope.go:117] "RemoveContainer" containerID="c9f3473874b1273398e39270d93674df76635c10b391568eb3a0b72b9fbe838b" Jul 10 18:39:36.299514 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:36.299488 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r" event={"ID":"fe032d86-7163-4e5b-9403-0b1804b79c1a","Type":"ContainerStarted","Data":"e7770c8e4f617fde6b5dedb5bab2d3e08689b9c630f0c138ff7f106bfbc1dba0"} Jul 10 18:39:36.300155 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:39:36.300134 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dw8rt\" is forbidden: User \"system:node:ip-10-0-140-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-180.ec2.internal' and this object" podUID="130ce75b-083d-4425-9407-e0b93e05c8f8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" Jul 10 18:39:36.301880 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:39:36.301854 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dw8rt\" is forbidden: User \"system:node:ip-10-0-140-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-180.ec2.internal' and this object" podUID="130ce75b-083d-4425-9407-e0b93e05c8f8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" Jul 10 18:39:36.306503 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:36.306489 2576 scope.go:117] "RemoveContainer" containerID="c9f3473874b1273398e39270d93674df76635c10b391568eb3a0b72b9fbe838b" Jul 10 18:39:36.306733 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:39:36.306715 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f3473874b1273398e39270d93674df76635c10b391568eb3a0b72b9fbe838b\": container with ID starting with c9f3473874b1273398e39270d93674df76635c10b391568eb3a0b72b9fbe838b not found: ID does not exist" containerID="c9f3473874b1273398e39270d93674df76635c10b391568eb3a0b72b9fbe838b" Jul 10 18:39:36.306789 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:36.306740 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f3473874b1273398e39270d93674df76635c10b391568eb3a0b72b9fbe838b"} err="failed to get container status \"c9f3473874b1273398e39270d93674df76635c10b391568eb3a0b72b9fbe838b\": rpc error: code = NotFound desc = could not find container \"c9f3473874b1273398e39270d93674df76635c10b391568eb3a0b72b9fbe838b\": container with ID starting with c9f3473874b1273398e39270d93674df76635c10b391568eb3a0b72b9fbe838b not found: ID does not exist" Jul 10 18:39:36.316713 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:36.316668 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zw5r" podStartSLOduration=1.352668612 podStartE2EDuration="24.316656698s" totalImagesPullingTime="22.963988086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:36.31630927 +0000 UTC m=+622.994021733" watchObservedRunningTime="2026-07-10 18:39:36.316656698 +0000 UTC m=+622.994369161" Jul 10 18:39:36.318052 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:39:36.318031 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dw8rt\" is forbidden: User \"system:node:ip-10-0-140-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-180.ec2.internal' and this object" podUID="130ce75b-083d-4425-9407-e0b93e05c8f8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dw8rt" Jul 10 18:39:36.355165 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:36.355136 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcpng\" (UniqueName: \"kubernetes.io/projected/130ce75b-083d-4425-9407-e0b93e05c8f8-kube-api-access-fcpng\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:39:37.844894 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:37.844857 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="130ce75b-083d-4425-9407-e0b93e05c8f8" path="/var/lib/kubelet/pods/130ce75b-083d-4425-9407-e0b93e05c8f8/volumes" Jul 10 18:39:38.518853 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:39:38.518801 2576 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-kjpv4_kuadrant-system_68a73088-fb31-4934-a10c-092d42453ff5_0(61d015129baa6079aecac47d9f8028212e2c92b42dae314399384b8a1597d39c): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-kjpv4 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 10 18:39:38.518984 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:39:38.518885 2576 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-kjpv4_kuadrant-system_68a73088-fb31-4934-a10c-092d42453ff5_0(61d015129baa6079aecac47d9f8028212e2c92b42dae314399384b8a1597d39c): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-kjpv4 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4" Jul 10 18:39:39.317381 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:39.317331 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4" Jul 10 18:39:39.319825 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:39:39.319783 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kjpv4\" is forbidden: User \"system:node:ip-10-0-140-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-180.ec2.internal' and this object" podUID="68a73088-fb31-4934-a10c-092d42453ff5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4" Jul 10 18:39:39.323687 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:39.323660 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4" Jul 10 18:39:39.325792 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:39:39.325763 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kjpv4\" is forbidden: User \"system:node:ip-10-0-140-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-180.ec2.internal' and this object" podUID="68a73088-fb31-4934-a10c-092d42453ff5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4" Jul 10 18:39:39.381165 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:39.381132 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68a73088-fb31-4934-a10c-092d42453ff5-extensions-socket-volume\") pod \"68a73088-fb31-4934-a10c-092d42453ff5\" (UID: \"68a73088-fb31-4934-a10c-092d42453ff5\") " Jul 10 18:39:39.381330 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:39.381227 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzpd4\" (UniqueName: \"kubernetes.io/projected/68a73088-fb31-4934-a10c-092d42453ff5-kube-api-access-mzpd4\") pod \"68a73088-fb31-4934-a10c-092d42453ff5\" (UID: \"68a73088-fb31-4934-a10c-092d42453ff5\") " Jul 10 18:39:39.381456 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:39.381431 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68a73088-fb31-4934-a10c-092d42453ff5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "68a73088-fb31-4934-a10c-092d42453ff5" (UID: "68a73088-fb31-4934-a10c-092d42453ff5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:39:39.383649 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:39.383623 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a73088-fb31-4934-a10c-092d42453ff5-kube-api-access-mzpd4" (OuterVolumeSpecName: "kube-api-access-mzpd4") pod "68a73088-fb31-4934-a10c-092d42453ff5" (UID: "68a73088-fb31-4934-a10c-092d42453ff5"). InnerVolumeSpecName "kube-api-access-mzpd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:39:39.482048 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:39.482008 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzpd4\" (UniqueName: \"kubernetes.io/projected/68a73088-fb31-4934-a10c-092d42453ff5-kube-api-access-mzpd4\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:39:39.482048 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:39.482049 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68a73088-fb31-4934-a10c-092d42453ff5-extensions-socket-volume\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:39:39.845710 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:39.845670 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68a73088-fb31-4934-a10c-092d42453ff5" path="/var/lib/kubelet/pods/68a73088-fb31-4934-a10c-092d42453ff5/volumes" Jul 10 18:39:40.324009 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:40.323965 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" event={"ID":"f471da5e-2233-4cee-83d8-b17b52006e70","Type":"ContainerStarted","Data":"5dd3587a8d22dc7b8e56d9465d500802fa9f90d8d80ea17766d6b05a019158c7"} Jul 10 18:39:40.324406 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:40.323980 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4" Jul 10 18:39:40.324406 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:40.324074 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" Jul 10 18:39:40.346443 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:40.346375 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" podStartSLOduration=13.375920628 podStartE2EDuration="17.346343607s" totalImagesPullingTime="3.970422979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:40.346202587 +0000 UTC m=+627.023915050" watchObservedRunningTime="2026-07-10 18:39:40.346343607 +0000 UTC m=+627.024056071" Jul 10 18:39:40.348520 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:39:40.348484 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kjpv4\" is forbidden: User \"system:node:ip-10-0-140-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-180.ec2.internal' and this object" podUID="68a73088-fb31-4934-a10c-092d42453ff5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kjpv4" Jul 10 18:39:51.331921 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:51.331887 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" Jul 10 18:39:51.674819 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:51.674740 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z"] Jul 10 18:39:51.675094 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:51.675082 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130ce75b-083d-4425-9407-e0b93e05c8f8" containerName="manager" Jul 10 18:39:51.675139 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:51.675096 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="130ce75b-083d-4425-9407-e0b93e05c8f8" containerName="manager" Jul 10 18:39:51.675172 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:51.675148 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="130ce75b-083d-4425-9407-e0b93e05c8f8" containerName="manager" Jul 10 18:39:51.692529 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:51.692500 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" Jul 10 18:39:51.695024 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:51.694998 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z"] Jul 10 18:39:51.784518 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:51.784488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce8605f3-2828-486c-99f6-8ebdae59b054-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rmt2z\" (UID: \"ce8605f3-2828-486c-99f6-8ebdae59b054\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" Jul 10 18:39:51.784729 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:51.784568 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v4bz\" (UniqueName: \"kubernetes.io/projected/ce8605f3-2828-486c-99f6-8ebdae59b054-kube-api-access-6v4bz\") pod \"kuadrant-operator-controller-manager-86878c769f-rmt2z\" (UID: \"ce8605f3-2828-486c-99f6-8ebdae59b054\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" Jul 10 18:39:51.885676 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:51.885643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v4bz\" (UniqueName: \"kubernetes.io/projected/ce8605f3-2828-486c-99f6-8ebdae59b054-kube-api-access-6v4bz\") pod \"kuadrant-operator-controller-manager-86878c769f-rmt2z\" (UID: \"ce8605f3-2828-486c-99f6-8ebdae59b054\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" Jul 10 18:39:51.885851 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:51.885701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce8605f3-2828-486c-99f6-8ebdae59b054-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rmt2z\" (UID: \"ce8605f3-2828-486c-99f6-8ebdae59b054\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" Jul 10 18:39:51.886090 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:51.886069 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce8605f3-2828-486c-99f6-8ebdae59b054-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rmt2z\" (UID: \"ce8605f3-2828-486c-99f6-8ebdae59b054\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" Jul 10 18:39:51.894410 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:51.894379 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v4bz\" (UniqueName: \"kubernetes.io/projected/ce8605f3-2828-486c-99f6-8ebdae59b054-kube-api-access-6v4bz\") pod \"kuadrant-operator-controller-manager-86878c769f-rmt2z\" (UID: \"ce8605f3-2828-486c-99f6-8ebdae59b054\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" Jul 10 18:39:52.003558 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:52.003467 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" Jul 10 18:39:52.135864 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:52.135841 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z"] Jul 10 18:39:52.138640 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:39:52.138614 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8605f3_2828_486c_99f6_8ebdae59b054.slice/crio-7052e8848851debea9914ba0194c1eac6d3c69804719cd05bee7edd86058559e WatchSource:0}: Error finding container 7052e8848851debea9914ba0194c1eac6d3c69804719cd05bee7edd86058559e: Status 404 returned error can't find the container with id 7052e8848851debea9914ba0194c1eac6d3c69804719cd05bee7edd86058559e Jul 10 18:39:52.386406 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:52.386364 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" event={"ID":"ce8605f3-2828-486c-99f6-8ebdae59b054","Type":"ContainerStarted","Data":"2e2121bd7e136f1a885d28e49bd36752ae5c75dbca8ba73c3f9004600971ba42"} Jul 10 18:39:52.386807 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:52.386425 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" event={"ID":"ce8605f3-2828-486c-99f6-8ebdae59b054","Type":"ContainerStarted","Data":"7052e8848851debea9914ba0194c1eac6d3c69804719cd05bee7edd86058559e"} Jul 10 18:39:52.386807 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:52.386453 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" Jul 10 18:39:52.409259 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:39:52.409207 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" podStartSLOduration=1.409192008 podStartE2EDuration="1.409192008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:52.407090548 +0000 UTC m=+639.084803010" watchObservedRunningTime="2026-07-10 18:39:52.409192008 +0000 UTC m=+639.086904470" Jul 10 18:40:03.393374 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:03.393326 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" Jul 10 18:40:03.446268 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:03.446237 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2"] Jul 10 18:40:03.446505 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:03.446457 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" podUID="f471da5e-2233-4cee-83d8-b17b52006e70" containerName="manager" containerID="cri-o://5dd3587a8d22dc7b8e56d9465d500802fa9f90d8d80ea17766d6b05a019158c7" gracePeriod=10 Jul 10 18:40:03.691509 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:03.691486 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" Jul 10 18:40:03.784776 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:03.784745 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f471da5e-2233-4cee-83d8-b17b52006e70-extensions-socket-volume\") pod \"f471da5e-2233-4cee-83d8-b17b52006e70\" (UID: \"f471da5e-2233-4cee-83d8-b17b52006e70\") " Jul 10 18:40:03.784959 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:03.784835 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv6km\" (UniqueName: \"kubernetes.io/projected/f471da5e-2233-4cee-83d8-b17b52006e70-kube-api-access-tv6km\") pod \"f471da5e-2233-4cee-83d8-b17b52006e70\" (UID: \"f471da5e-2233-4cee-83d8-b17b52006e70\") " Jul 10 18:40:03.785216 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:03.785191 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f471da5e-2233-4cee-83d8-b17b52006e70-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f471da5e-2233-4cee-83d8-b17b52006e70" (UID: "f471da5e-2233-4cee-83d8-b17b52006e70"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:40:03.786852 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:03.786830 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f471da5e-2233-4cee-83d8-b17b52006e70-kube-api-access-tv6km" (OuterVolumeSpecName: "kube-api-access-tv6km") pod "f471da5e-2233-4cee-83d8-b17b52006e70" (UID: "f471da5e-2233-4cee-83d8-b17b52006e70"). InnerVolumeSpecName "kube-api-access-tv6km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:40:03.885699 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:03.885674 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv6km\" (UniqueName: \"kubernetes.io/projected/f471da5e-2233-4cee-83d8-b17b52006e70-kube-api-access-tv6km\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:40:03.885699 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:03.885697 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f471da5e-2233-4cee-83d8-b17b52006e70-extensions-socket-volume\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:40:04.452873 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:04.452834 2576 generic.go:356] "Generic (PLEG): container finished" podID="f471da5e-2233-4cee-83d8-b17b52006e70" containerID="5dd3587a8d22dc7b8e56d9465d500802fa9f90d8d80ea17766d6b05a019158c7" exitCode=0 Jul 10 18:40:04.453284 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:04.452883 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" Jul 10 18:40:04.453284 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:04.452917 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" event={"ID":"f471da5e-2233-4cee-83d8-b17b52006e70","Type":"ContainerDied","Data":"5dd3587a8d22dc7b8e56d9465d500802fa9f90d8d80ea17766d6b05a019158c7"} Jul 10 18:40:04.453284 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:04.452975 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2" event={"ID":"f471da5e-2233-4cee-83d8-b17b52006e70","Type":"ContainerDied","Data":"ad85af2f82bd69ee222ac498e53bbf371f1ccb6886bced21792eb20df1e14b93"} Jul 10 18:40:04.453284 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:04.452998 2576 scope.go:117] "RemoveContainer" containerID="5dd3587a8d22dc7b8e56d9465d500802fa9f90d8d80ea17766d6b05a019158c7" Jul 10 18:40:04.461726 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:04.461707 2576 scope.go:117] "RemoveContainer" containerID="5dd3587a8d22dc7b8e56d9465d500802fa9f90d8d80ea17766d6b05a019158c7" Jul 10 18:40:04.461997 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:40:04.461973 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dd3587a8d22dc7b8e56d9465d500802fa9f90d8d80ea17766d6b05a019158c7\": container with ID starting with 5dd3587a8d22dc7b8e56d9465d500802fa9f90d8d80ea17766d6b05a019158c7 not found: ID does not exist" containerID="5dd3587a8d22dc7b8e56d9465d500802fa9f90d8d80ea17766d6b05a019158c7" Jul 10 18:40:04.462054 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:04.461999 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd3587a8d22dc7b8e56d9465d500802fa9f90d8d80ea17766d6b05a019158c7"} err="failed to get container status \"5dd3587a8d22dc7b8e56d9465d500802fa9f90d8d80ea17766d6b05a019158c7\": rpc error: code = NotFound desc = could not find container \"5dd3587a8d22dc7b8e56d9465d500802fa9f90d8d80ea17766d6b05a019158c7\": container with ID starting with 5dd3587a8d22dc7b8e56d9465d500802fa9f90d8d80ea17766d6b05a019158c7 not found: ID does not exist" Jul 10 18:40:04.471610 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:04.471585 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2"] Jul 10 18:40:04.478147 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:04.478126 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zhqt2"] Jul 10 18:40:05.842879 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:05.842843 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f471da5e-2233-4cee-83d8-b17b52006e70" path="/var/lib/kubelet/pods/f471da5e-2233-4cee-83d8-b17b52006e70/volumes" Jul 10 18:40:07.622207 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.622178 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq"] Jul 10 18:40:07.622604 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.622552 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f471da5e-2233-4cee-83d8-b17b52006e70" containerName="manager" Jul 10 18:40:07.622604 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.622564 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f471da5e-2233-4cee-83d8-b17b52006e70" containerName="manager" Jul 10 18:40:07.622679 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.622618 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f471da5e-2233-4cee-83d8-b17b52006e70" containerName="manager" Jul 10 18:40:07.664439 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.664392 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq"] Jul 10 18:40:07.664624 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.664483 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.666769 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.666746 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nkbxz\"" Jul 10 18:40:07.717420 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.717386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x26lp\" (UniqueName: \"kubernetes.io/projected/f3d5f957-c477-465a-8199-48833698b766-kube-api-access-x26lp\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.717617 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.717437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d5f957-c477-465a-8199-48833698b766-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.717617 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.717534 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d5f957-c477-465a-8199-48833698b766-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.717617 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.717588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d5f957-c477-465a-8199-48833698b766-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.717793 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.717651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d5f957-c477-465a-8199-48833698b766-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.717793 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.717676 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d5f957-c477-465a-8199-48833698b766-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.717793 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.717724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d5f957-c477-465a-8199-48833698b766-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.717793 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.717754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d5f957-c477-465a-8199-48833698b766-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.717793 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.717788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d5f957-c477-465a-8199-48833698b766-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.819143 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.819105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d5f957-c477-465a-8199-48833698b766-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.819143 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.819142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d5f957-c477-465a-8199-48833698b766-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.819425 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.819168 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d5f957-c477-465a-8199-48833698b766-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.819425 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.819189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d5f957-c477-465a-8199-48833698b766-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.819425 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.819216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d5f957-c477-465a-8199-48833698b766-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.819425 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.819250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26lp\" (UniqueName: \"kubernetes.io/projected/f3d5f957-c477-465a-8199-48833698b766-kube-api-access-x26lp\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.819425 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.819288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d5f957-c477-465a-8199-48833698b766-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.819425 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.819329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d5f957-c477-465a-8199-48833698b766-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.819425 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.819391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d5f957-c477-465a-8199-48833698b766-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.819767 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.819655 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d5f957-c477-465a-8199-48833698b766-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.819803 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.819787 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d5f957-c477-465a-8199-48833698b766-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.819970 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.819950 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d5f957-c477-465a-8199-48833698b766-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.820047 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.819994 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d5f957-c477-465a-8199-48833698b766-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.820435 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.820414 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d5f957-c477-465a-8199-48833698b766-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.821521 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.821496 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d5f957-c477-465a-8199-48833698b766-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.821881 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.821853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d5f957-c477-465a-8199-48833698b766-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.827591 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.827568 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d5f957-c477-465a-8199-48833698b766-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.827699 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.827575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x26lp\" (UniqueName: \"kubernetes.io/projected/f3d5f957-c477-465a-8199-48833698b766-kube-api-access-x26lp\") pod \"maas-default-gateway-openshift-default-5896d4dd49-npkgq\" (UID: \"f3d5f957-c477-465a-8199-48833698b766\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:07.981189 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:07.981095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:08.130479 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:08.130446 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq"] Jul 10 18:40:08.131303 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:40:08.131272 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d5f957_c477_465a_8199_48833698b766.slice/crio-01d05747e42f8f70ae88b14bbeb1a3d1a99a3e05d69a428ae22b2cb80fb9734c WatchSource:0}: Error finding container 01d05747e42f8f70ae88b14bbeb1a3d1a99a3e05d69a428ae22b2cb80fb9734c: Status 404 returned error can't find the container with id 01d05747e42f8f70ae88b14bbeb1a3d1a99a3e05d69a428ae22b2cb80fb9734c Jul 10 18:40:08.133300 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:08.133257 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:40:08.133398 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:08.133329 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:40:08.133398 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:08.133383 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:40:08.476698 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:08.476657 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" event={"ID":"f3d5f957-c477-465a-8199-48833698b766","Type":"ContainerStarted","Data":"5156bf13c7a837444e681f5e2ef40e39db95aa6d29e0f72c3e68fe0a06fe47cd"} Jul 10 18:40:08.476867 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:08.476717 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" event={"ID":"f3d5f957-c477-465a-8199-48833698b766","Type":"ContainerStarted","Data":"01d05747e42f8f70ae88b14bbeb1a3d1a99a3e05d69a428ae22b2cb80fb9734c"} Jul 10 18:40:08.496989 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:08.496933 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" podStartSLOduration=1.4969178109999999 podStartE2EDuration="1.496917811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:08.49503619 +0000 UTC m=+655.172748652" watchObservedRunningTime="2026-07-10 18:40:08.496917811 +0000 UTC m=+655.174630273" Jul 10 18:40:08.981724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:08.981696 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:09.986255 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:09.986225 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:10.487295 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:10.487267 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:10.488150 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:10.488131 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:40:11.022905 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:11.022868 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:40:11.025638 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:11.025618 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-hnt6h" Jul 10 18:40:11.027609 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:11.027584 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 18:40:11.035521 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:11.035498 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:40:11.061222 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:11.061198 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:40:11.151542 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:11.151511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2859ebf-9ede-4d35-bf81-34d56d12c5c7-config-file\") pod \"limitador-limitador-54d6d988d5-hnt6h\" (UID: \"a2859ebf-9ede-4d35-bf81-34d56d12c5c7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hnt6h" Jul 10 18:40:11.151699 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:11.151556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-729l9\" (UniqueName: \"kubernetes.io/projected/a2859ebf-9ede-4d35-bf81-34d56d12c5c7-kube-api-access-729l9\") pod \"limitador-limitador-54d6d988d5-hnt6h\" (UID: \"a2859ebf-9ede-4d35-bf81-34d56d12c5c7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hnt6h" Jul 10 18:40:11.252630 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:11.252582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2859ebf-9ede-4d35-bf81-34d56d12c5c7-config-file\") pod \"limitador-limitador-54d6d988d5-hnt6h\" (UID: \"a2859ebf-9ede-4d35-bf81-34d56d12c5c7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hnt6h" Jul 10 18:40:11.252811 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:11.252653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-729l9\" (UniqueName: \"kubernetes.io/projected/a2859ebf-9ede-4d35-bf81-34d56d12c5c7-kube-api-access-729l9\") pod \"limitador-limitador-54d6d988d5-hnt6h\" (UID: \"a2859ebf-9ede-4d35-bf81-34d56d12c5c7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hnt6h" Jul 10 18:40:11.253252 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:11.253225 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2859ebf-9ede-4d35-bf81-34d56d12c5c7-config-file\") pod \"limitador-limitador-54d6d988d5-hnt6h\" (UID: \"a2859ebf-9ede-4d35-bf81-34d56d12c5c7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hnt6h" Jul 10 18:40:11.260496 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:11.260464 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-729l9\" (UniqueName: \"kubernetes.io/projected/a2859ebf-9ede-4d35-bf81-34d56d12c5c7-kube-api-access-729l9\") pod \"limitador-limitador-54d6d988d5-hnt6h\" (UID: \"a2859ebf-9ede-4d35-bf81-34d56d12c5c7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-hnt6h" Jul 10 18:40:11.337143 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:11.337107 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-hnt6h" Jul 10 18:40:11.462764 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:11.462736 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:40:11.463924 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:40:11.463894 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2859ebf_9ede_4d35_bf81_34d56d12c5c7.slice/crio-ad416e175db87082940897992af8ca607b7ade68400dede26598396be64aafea WatchSource:0}: Error finding container ad416e175db87082940897992af8ca607b7ade68400dede26598396be64aafea: Status 404 returned error can't find the container with id ad416e175db87082940897992af8ca607b7ade68400dede26598396be64aafea Jul 10 18:40:11.500515 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:11.500479 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-hnt6h" event={"ID":"a2859ebf-9ede-4d35-bf81-34d56d12c5c7","Type":"ContainerStarted","Data":"ad416e175db87082940897992af8ca607b7ade68400dede26598396be64aafea"} Jul 10 18:40:15.525300 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:15.525261 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-hnt6h" event={"ID":"a2859ebf-9ede-4d35-bf81-34d56d12c5c7","Type":"ContainerStarted","Data":"36a47f0489e286763b2d860c08793227d721658ccc55a9114fa92cbedf277cbd"} Jul 10 18:40:15.525687 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:15.525368 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-hnt6h" Jul 10 18:40:15.542224 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:15.542167 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-hnt6h" podStartSLOduration=1.519565794 podStartE2EDuration="4.5421526s" totalImagesPullingTime="3.022586806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:15.541506481 +0000 UTC m=+662.219218944" watchObservedRunningTime="2026-07-10 18:40:15.5421526 +0000 UTC m=+662.219865105" Jul 10 18:40:26.531389 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:40:26.531332 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-hnt6h" Jul 10 18:41:27.610931 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:27.610836 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:41:34.925558 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:34.925452 2576 generic.go:356] "Generic (PLEG): container finished" podID="f3d5f957-c477-465a-8199-48833698b766" containerID="5156bf13c7a837444e681f5e2ef40e39db95aa6d29e0f72c3e68fe0a06fe47cd" exitCode=0 Jul 10 18:41:34.925558 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:34.925529 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" event={"ID":"f3d5f957-c477-465a-8199-48833698b766","Type":"ContainerDied","Data":"5156bf13c7a837444e681f5e2ef40e39db95aa6d29e0f72c3e68fe0a06fe47cd"} Jul 10 18:41:34.926100 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:34.926006 2576 scope.go:117] "RemoveContainer" containerID="5156bf13c7a837444e681f5e2ef40e39db95aa6d29e0f72c3e68fe0a06fe47cd" Jul 10 18:41:34.926967 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:34.926938 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:41:34.927073 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:34.927013 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:41:34.927073 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:34.927059 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:41:35.934420 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:35.933528 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" event={"ID":"f3d5f957-c477-465a-8199-48833698b766","Type":"ContainerStarted","Data":"9c163941a0f68e3c3445c6ecb71ac0f1f67291a3d5f2450d883531f35fd409d3"} Jul 10 18:41:35.981766 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:35.981735 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:41:35.982766 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:35.982746 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-npkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 10 18:41:35.982840 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:35.982791 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" podUID="f3d5f957-c477-465a-8199-48833698b766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 10 18:41:36.981699 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:36.981667 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-npkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 10 18:41:36.982106 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:36.981719 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" podUID="f3d5f957-c477-465a-8199-48833698b766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 10 18:41:37.981485 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:37.981450 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-npkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 10 18:41:37.981666 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:37.981504 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" podUID="f3d5f957-c477-465a-8199-48833698b766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 10 18:41:38.981830 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:38.981796 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-npkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 10 18:41:38.982304 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:38.981866 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" podUID="f3d5f957-c477-465a-8199-48833698b766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 10 18:41:40.029336 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:40.029303 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:41:40.029848 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:40.029750 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:41:40.030582 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:40.030559 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-npkgq" Jul 10 18:41:44.034830 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:41:44.034787 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:42:08.087846 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:08.087808 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:42:13.090719 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:13.090687 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:42:27.791770 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:27.791725 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:42:29.186107 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.186073 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds"] Jul 10 18:42:29.189869 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.189850 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.192855 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.192824 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 18:42:29.193156 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.193141 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 18:42:29.193377 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.193363 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-69xjn\"" Jul 10 18:42:29.193998 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.193984 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 18:42:29.203402 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.203382 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds"] Jul 10 18:42:29.304989 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.304957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpjxh\" (UniqueName: \"kubernetes.io/projected/28051884-0ba2-40a4-82c9-679d77da1252-kube-api-access-mpjxh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.305162 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.304997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28051884-0ba2-40a4-82c9-679d77da1252-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.305162 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.305022 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28051884-0ba2-40a4-82c9-679d77da1252-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.305162 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.305068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28051884-0ba2-40a4-82c9-679d77da1252-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.305162 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.305120 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28051884-0ba2-40a4-82c9-679d77da1252-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.305330 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.305170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28051884-0ba2-40a4-82c9-679d77da1252-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.406304 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.406263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28051884-0ba2-40a4-82c9-679d77da1252-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.406304 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.406307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28051884-0ba2-40a4-82c9-679d77da1252-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.406578 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.406330 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28051884-0ba2-40a4-82c9-679d77da1252-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.406578 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.406532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28051884-0ba2-40a4-82c9-679d77da1252-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.406695 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.406603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28051884-0ba2-40a4-82c9-679d77da1252-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.406695 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.406669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpjxh\" (UniqueName: \"kubernetes.io/projected/28051884-0ba2-40a4-82c9-679d77da1252-kube-api-access-mpjxh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.406793 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.406671 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28051884-0ba2-40a4-82c9-679d77da1252-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.406793 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.406730 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28051884-0ba2-40a4-82c9-679d77da1252-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.406932 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.406904 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28051884-0ba2-40a4-82c9-679d77da1252-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.408750 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.408716 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28051884-0ba2-40a4-82c9-679d77da1252-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.408982 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.408967 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28051884-0ba2-40a4-82c9-679d77da1252-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.417189 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.417167 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpjxh\" (UniqueName: \"kubernetes.io/projected/28051884-0ba2-40a4-82c9-679d77da1252-kube-api-access-mpjxh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lrds\" (UID: \"28051884-0ba2-40a4-82c9-679d77da1252\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.501310 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.501214 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:29.636600 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.636578 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds"] Jul 10 18:42:29.638640 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:42:29.638617 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28051884_0ba2_40a4_82c9_679d77da1252.slice/crio-24ebedff119bced44059a36664986c05b04dd05edb492642d9f269a62bff7135 WatchSource:0}: Error finding container 24ebedff119bced44059a36664986c05b04dd05edb492642d9f269a62bff7135: Status 404 returned error can't find the container with id 24ebedff119bced44059a36664986c05b04dd05edb492642d9f269a62bff7135 Jul 10 18:42:29.639926 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:29.639910 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 18:42:30.227440 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:30.227397 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" event={"ID":"28051884-0ba2-40a4-82c9-679d77da1252","Type":"ContainerStarted","Data":"24ebedff119bced44059a36664986c05b04dd05edb492642d9f269a62bff7135"} Jul 10 18:42:31.234292 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:31.234255 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" event={"ID":"28051884-0ba2-40a4-82c9-679d77da1252","Type":"ContainerStarted","Data":"0ac87260c5e43ce2a850dbcf41a04da71223898e08248aa27b54bb05f2a38ad7"} Jul 10 18:42:31.254175 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:31.254127 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" podStartSLOduration=1.255620808 podStartE2EDuration="2.254112849s" totalImagesPullingTime="998.492041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:42:31.251259845 +0000 UTC m=+797.928972345" watchObservedRunningTime="2026-07-10 18:42:31.254112849 +0000 UTC m=+797.931825344" Jul 10 18:42:33.688223 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.688191 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx"] Jul 10 18:42:33.691837 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.691814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.695001 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.694982 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 10 18:42:33.703989 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.703968 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx"] Jul 10 18:42:33.743169 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.743127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.743342 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.743204 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.743342 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.743284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.743485 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.743371 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.743485 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.743401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.743485 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.743430 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdmd4\" (UniqueName: \"kubernetes.io/projected/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-kube-api-access-sdmd4\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.844251 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.844215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.844454 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.844267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.844454 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.844303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.844454 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.844329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.844454 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.844368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmd4\" (UniqueName: \"kubernetes.io/projected/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-kube-api-access-sdmd4\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.844679 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.844455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.844771 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.844731 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.844943 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.844773 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.844943 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.844813 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.846775 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.846754 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.846857 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.846828 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:33.852465 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:33.852444 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdmd4\" (UniqueName: \"kubernetes.io/projected/2a96fd99-f457-4315-a7bb-5dc0bdc584b7-kube-api-access-sdmd4\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx\" (UID: \"2a96fd99-f457-4315-a7bb-5dc0bdc584b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:34.001289 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:34.001203 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:34.133399 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:34.133375 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx"] Jul 10 18:42:34.134955 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:42:34.134918 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a96fd99_f457_4315_a7bb_5dc0bdc584b7.slice/crio-5e0d44ca74bfb52c06ae46cee6cd9549e6408c66be4b87b4cc16f896b5238a5a WatchSource:0}: Error finding container 5e0d44ca74bfb52c06ae46cee6cd9549e6408c66be4b87b4cc16f896b5238a5a: Status 404 returned error can't find the container with id 5e0d44ca74bfb52c06ae46cee6cd9549e6408c66be4b87b4cc16f896b5238a5a Jul 10 18:42:34.250801 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:34.250767 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" event={"ID":"2a96fd99-f457-4315-a7bb-5dc0bdc584b7","Type":"ContainerStarted","Data":"5e0d44ca74bfb52c06ae46cee6cd9549e6408c66be4b87b4cc16f896b5238a5a"} Jul 10 18:42:35.257780 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:35.257745 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" event={"ID":"2a96fd99-f457-4315-a7bb-5dc0bdc584b7","Type":"ContainerStarted","Data":"453b91f5b2d9c067004cc937eaedccf80660dbd9f31a84dca7f6b6b607a06536"} Jul 10 18:42:35.275496 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:35.275441 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" podStartSLOduration=2.107926193 podStartE2EDuration="2.275421152s" totalImagesPullingTime="167.494959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:42:35.274236075 +0000 UTC m=+801.951948537" watchObservedRunningTime="2026-07-10 18:42:35.275421152 +0000 UTC m=+801.953133605" Jul 10 18:42:36.283619 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:36.283538 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:42:39.501458 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:39.501423 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:39.501960 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:39.501496 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:39.513672 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:39.513650 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:40.292085 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:40.292053 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lrds" Jul 10 18:42:44.001386 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:44.001331 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:44.001843 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:44.001409 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:44.015076 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:44.015054 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:42:44.311648 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:42:44.311568 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx" Jul 10 18:44:02.995827 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:44:02.995794 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:44:12.385823 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:44:12.385739 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:44:13.808602 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:44:13.808578 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:44:13.809669 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:44:13.809646 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:44:23.080493 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:44:23.080456 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:44:31.584438 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:44:31.584398 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:44:42.685237 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:44:42.685206 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:45:09.589043 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:45:09.589007 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:45:22.393016 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:45:22.392981 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:45:55.988251 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:45:55.988172 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:46:05.780486 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:46:05.780451 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:46:16.288444 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:46:16.288410 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:46:23.783038 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:46:23.783008 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:46:33.418316 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:46:33.418285 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:46:44.796371 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:46:44.796312 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:46:56.095678 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:46:56.095641 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:47:05.690095 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:47:05.690061 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:47:27.885566 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:47:27.885489 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:47:39.492932 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:47:39.492897 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:48:03.095791 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:48:03.095755 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:48:13.816412 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:48:13.816377 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:48:28.592859 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:48:28.592822 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:48:39.399386 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:48:39.399291 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:48:49.181489 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:48:49.181453 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:48:59.083174 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:48:59.083138 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:49:08.587443 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:08.587408 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:49:13.844619 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:13.844575 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:49:13.846180 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:13.846153 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:49:32.867857 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:32.867819 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z"] Jul 10 18:49:32.868275 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:32.868037 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" podUID="ce8605f3-2828-486c-99f6-8ebdae59b054" containerName="manager" containerID="cri-o://2e2121bd7e136f1a885d28e49bd36752ae5c75dbca8ba73c3f9004600971ba42" gracePeriod=10 Jul 10 18:49:35.318393 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.318368 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" Jul 10 18:49:35.432462 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.432342 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce8605f3-2828-486c-99f6-8ebdae59b054-extensions-socket-volume\") pod \"ce8605f3-2828-486c-99f6-8ebdae59b054\" (UID: \"ce8605f3-2828-486c-99f6-8ebdae59b054\") " Jul 10 18:49:35.432462 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.432424 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v4bz\" (UniqueName: \"kubernetes.io/projected/ce8605f3-2828-486c-99f6-8ebdae59b054-kube-api-access-6v4bz\") pod \"ce8605f3-2828-486c-99f6-8ebdae59b054\" (UID: \"ce8605f3-2828-486c-99f6-8ebdae59b054\") " Jul 10 18:49:35.432832 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.432806 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8605f3-2828-486c-99f6-8ebdae59b054-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ce8605f3-2828-486c-99f6-8ebdae59b054" (UID: "ce8605f3-2828-486c-99f6-8ebdae59b054"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:49:35.434548 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.434524 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8605f3-2828-486c-99f6-8ebdae59b054-kube-api-access-6v4bz" (OuterVolumeSpecName: "kube-api-access-6v4bz") pod "ce8605f3-2828-486c-99f6-8ebdae59b054" (UID: "ce8605f3-2828-486c-99f6-8ebdae59b054"). InnerVolumeSpecName "kube-api-access-6v4bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:49:35.445945 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.445917 2576 generic.go:356] "Generic (PLEG): container finished" podID="ce8605f3-2828-486c-99f6-8ebdae59b054" containerID="2e2121bd7e136f1a885d28e49bd36752ae5c75dbca8ba73c3f9004600971ba42" exitCode=0 Jul 10 18:49:35.446078 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.445976 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" Jul 10 18:49:35.446078 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.445987 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" event={"ID":"ce8605f3-2828-486c-99f6-8ebdae59b054","Type":"ContainerDied","Data":"2e2121bd7e136f1a885d28e49bd36752ae5c75dbca8ba73c3f9004600971ba42"} Jul 10 18:49:35.446078 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.446054 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z" event={"ID":"ce8605f3-2828-486c-99f6-8ebdae59b054","Type":"ContainerDied","Data":"7052e8848851debea9914ba0194c1eac6d3c69804719cd05bee7edd86058559e"} Jul 10 18:49:35.446078 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.446077 2576 scope.go:117] "RemoveContainer" containerID="2e2121bd7e136f1a885d28e49bd36752ae5c75dbca8ba73c3f9004600971ba42" Jul 10 18:49:35.455764 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.455747 2576 scope.go:117] "RemoveContainer" containerID="2e2121bd7e136f1a885d28e49bd36752ae5c75dbca8ba73c3f9004600971ba42" Jul 10 18:49:35.456019 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:49:35.456002 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2121bd7e136f1a885d28e49bd36752ae5c75dbca8ba73c3f9004600971ba42\": container with ID starting with 2e2121bd7e136f1a885d28e49bd36752ae5c75dbca8ba73c3f9004600971ba42 not found: ID does not exist" containerID="2e2121bd7e136f1a885d28e49bd36752ae5c75dbca8ba73c3f9004600971ba42" Jul 10 18:49:35.456061 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.456026 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2121bd7e136f1a885d28e49bd36752ae5c75dbca8ba73c3f9004600971ba42"} err="failed to get container status \"2e2121bd7e136f1a885d28e49bd36752ae5c75dbca8ba73c3f9004600971ba42\": rpc error: code = NotFound desc = could not find container \"2e2121bd7e136f1a885d28e49bd36752ae5c75dbca8ba73c3f9004600971ba42\": container with ID starting with 2e2121bd7e136f1a885d28e49bd36752ae5c75dbca8ba73c3f9004600971ba42 not found: ID does not exist" Jul 10 18:49:35.467724 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.467691 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z"] Jul 10 18:49:35.471412 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.471389 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rmt2z"] Jul 10 18:49:35.533747 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.533713 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v4bz\" (UniqueName: \"kubernetes.io/projected/ce8605f3-2828-486c-99f6-8ebdae59b054-kube-api-access-6v4bz\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:49:35.533747 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.533742 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce8605f3-2828-486c-99f6-8ebdae59b054-extensions-socket-volume\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:49:35.843274 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:49:35.843242 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce8605f3-2828-486c-99f6-8ebdae59b054" path="/var/lib/kubelet/pods/ce8605f3-2828-486c-99f6-8ebdae59b054/volumes" Jul 10 18:50:33.972860 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:33.972825 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n"] Jul 10 18:50:33.973288 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:33.973185 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce8605f3-2828-486c-99f6-8ebdae59b054" containerName="manager" Jul 10 18:50:33.973288 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:33.973197 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce8605f3-2828-486c-99f6-8ebdae59b054" containerName="manager" Jul 10 18:50:33.973288 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:33.973263 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce8605f3-2828-486c-99f6-8ebdae59b054" containerName="manager" Jul 10 18:50:33.979446 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:33.979410 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n" Jul 10 18:50:33.982005 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:33.981970 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x9lc4\"" Jul 10 18:50:33.986131 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:33.986110 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n"] Jul 10 18:50:34.026957 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:34.026927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnn4\" (UniqueName: \"kubernetes.io/projected/28d2da83-2006-4378-bd15-4194b7b2b13f-kube-api-access-flnn4\") pod \"kuadrant-operator-controller-manager-86878c769f-k425n\" (UID: \"28d2da83-2006-4378-bd15-4194b7b2b13f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n" Jul 10 18:50:34.027077 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:34.026978 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28d2da83-2006-4378-bd15-4194b7b2b13f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k425n\" (UID: \"28d2da83-2006-4378-bd15-4194b7b2b13f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n" Jul 10 18:50:34.127382 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:34.127341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28d2da83-2006-4378-bd15-4194b7b2b13f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k425n\" (UID: \"28d2da83-2006-4378-bd15-4194b7b2b13f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n" Jul 10 18:50:34.127547 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:34.127423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnn4\" (UniqueName: \"kubernetes.io/projected/28d2da83-2006-4378-bd15-4194b7b2b13f-kube-api-access-flnn4\") pod \"kuadrant-operator-controller-manager-86878c769f-k425n\" (UID: \"28d2da83-2006-4378-bd15-4194b7b2b13f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n" Jul 10 18:50:34.127768 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:34.127746 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28d2da83-2006-4378-bd15-4194b7b2b13f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k425n\" (UID: \"28d2da83-2006-4378-bd15-4194b7b2b13f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n" Jul 10 18:50:34.140033 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:34.140009 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnn4\" (UniqueName: \"kubernetes.io/projected/28d2da83-2006-4378-bd15-4194b7b2b13f-kube-api-access-flnn4\") pod \"kuadrant-operator-controller-manager-86878c769f-k425n\" (UID: \"28d2da83-2006-4378-bd15-4194b7b2b13f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n" Jul 10 18:50:34.291839 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:34.291747 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n" Jul 10 18:50:34.428518 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:34.428492 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n"] Jul 10 18:50:34.430593 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:50:34.430559 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d2da83_2006_4378_bd15_4194b7b2b13f.slice/crio-397a08484b6bbeb8f9037aa859b3d430b9baed7f184b5c8696a65e323bc684db WatchSource:0}: Error finding container 397a08484b6bbeb8f9037aa859b3d430b9baed7f184b5c8696a65e323bc684db: Status 404 returned error can't find the container with id 397a08484b6bbeb8f9037aa859b3d430b9baed7f184b5c8696a65e323bc684db Jul 10 18:50:34.746375 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:34.746324 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n" event={"ID":"28d2da83-2006-4378-bd15-4194b7b2b13f","Type":"ContainerStarted","Data":"3ed75da5af0b4cffb9717545021d176ba090982365e342184629997b17384faa"} Jul 10 18:50:34.746529 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:34.746402 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n" event={"ID":"28d2da83-2006-4378-bd15-4194b7b2b13f","Type":"ContainerStarted","Data":"397a08484b6bbeb8f9037aa859b3d430b9baed7f184b5c8696a65e323bc684db"} Jul 10 18:50:34.746529 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:34.746435 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n" Jul 10 18:50:34.766750 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:34.766704 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n" podStartSLOduration=1.7666855030000002 podStartE2EDuration="1.766685503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:50:34.764664479 +0000 UTC m=+1281.442376941" watchObservedRunningTime="2026-07-10 18:50:34.766685503 +0000 UTC m=+1281.444397966" Jul 10 18:50:45.755884 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:45.755855 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k425n" Jul 10 18:50:54.292407 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:50:54.292368 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:51:04.382298 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:51:04.382263 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:51:16.585530 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:51:16.585496 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:51:28.482889 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:51:28.482839 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:51:37.992663 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:51:37.992585 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:51:49.389247 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:51:49.389209 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:51:58.693001 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:51:58.692966 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:52:09.889663 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:52:09.889626 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:52:20.540240 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:52:20.540201 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:52:29.192610 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:52:29.192576 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:52:39.688227 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:52:39.688189 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:53:02.165855 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:53:02.165817 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:53:11.615080 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:53:11.615001 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:53:21.894167 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:53:21.894130 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:53:30.903799 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:53:30.903760 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:53:38.095410 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:53:38.095376 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:53:54.298543 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:53:54.298500 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:54:04.092699 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:54:04.092660 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:54:13.401664 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:54:13.401627 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:54:13.879417 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:54:13.879387 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:54:13.881772 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:54:13.881743 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:54:23.282310 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:54:23.282274 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:54:34.596492 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:54:34.596459 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:54:44.182905 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:54:44.182821 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:54:56.887863 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:54:56.887826 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:55:05.987329 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:55:05.987295 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:55:15.883371 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:55:15.883318 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:55:28.193841 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:55:28.193804 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:55:39.694024 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:55:39.693990 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:55:48.495284 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:55:48.495240 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:55:58.895642 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:55:58.895604 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:56:15.584688 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:15.584609 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:56:27.185450 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:27.185417 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:56:37.983309 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:37.983269 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:56:48.690147 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:48.690115 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:56:57.929432 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:57.929397 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg"] Jul 10 18:56:57.933071 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:57.933050 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:57.937542 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:57.937519 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d487a3ed-openshift-default-dockercfg-s4vkj\"" Jul 10 18:56:57.959501 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:57.959477 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg"] Jul 10 18:56:58.022111 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.022077 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-data\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.022111 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.022112 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ecad98-b686-4f6c-a045-13ff5fedd39f-istiod-ca-cert\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.022308 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.022131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-credential-socket\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.022308 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.022182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fksr\" (UniqueName: \"kubernetes.io/projected/23ecad98-b686-4f6c-a045-13ff5fedd39f-kube-api-access-7fksr\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.022308 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.022218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-token\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.022308 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.022260 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-workload-certs\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.022308 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.022295 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-workload-socket\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.022491 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.022311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-podinfo\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.022491 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.022394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-envoy\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.123082 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.123052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-envoy\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.123082 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.123089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-data\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.123283 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.123105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ecad98-b686-4f6c-a045-13ff5fedd39f-istiod-ca-cert\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.123283 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.123122 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-credential-socket\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.123283 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.123262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fksr\" (UniqueName: \"kubernetes.io/projected/23ecad98-b686-4f6c-a045-13ff5fedd39f-kube-api-access-7fksr\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.123440 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.123308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-token\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.123440 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.123407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-workload-certs\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.123547 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.123456 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-workload-socket\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.123547 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.123480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-podinfo\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.123547 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.123525 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-credential-socket\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.123864 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.123844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-data\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.123950 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.123885 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ecad98-b686-4f6c-a045-13ff5fedd39f-istiod-ca-cert\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.123986 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.123957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-workload-certs\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.124020 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.123997 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-workload-socket\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.125475 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.125458 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-envoy\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.125880 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.125865 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-podinfo\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.131942 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.131913 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-token\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.132078 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.132057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fksr\" (UniqueName: \"kubernetes.io/projected/23ecad98-b686-4f6c-a045-13ff5fedd39f-kube-api-access-7fksr\") pod \"e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.243537 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.243451 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:58.379101 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.378939 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg"] Jul 10 18:56:58.381302 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:56:58.381277 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ecad98_b686_4f6c_a045_13ff5fedd39f.slice/crio-ee2668e32211b12bf94536c008ec886bafa55f08159b3bcba2e7b7777548dc22 WatchSource:0}: Error finding container ee2668e32211b12bf94536c008ec886bafa55f08159b3bcba2e7b7777548dc22: Status 404 returned error can't find the container with id ee2668e32211b12bf94536c008ec886bafa55f08159b3bcba2e7b7777548dc22 Jul 10 18:56:58.383331 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.383300 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:56:58.383436 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.383392 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:56:58.383436 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.383424 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:56:58.489228 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.489198 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:56:58.720954 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.720917 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" event={"ID":"23ecad98-b686-4f6c-a045-13ff5fedd39f","Type":"ContainerStarted","Data":"846c1a4a1038e65e761bdcde112e61a95f013c4be305cf7f651390c392a823fe"} Jul 10 18:56:58.721151 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.720972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" event={"ID":"23ecad98-b686-4f6c-a045-13ff5fedd39f","Type":"ContainerStarted","Data":"ee2668e32211b12bf94536c008ec886bafa55f08159b3bcba2e7b7777548dc22"} Jul 10 18:56:58.749233 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:58.749174 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" podStartSLOduration=1.7491572720000002 podStartE2EDuration="1.749157272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:56:58.746853689 +0000 UTC m=+1665.424566153" watchObservedRunningTime="2026-07-10 18:56:58.749157272 +0000 UTC m=+1665.426869725" Jul 10 18:56:59.244533 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:59.244486 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:59.249602 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:59.249576 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:59.726183 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:59.726150 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:56:59.727217 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:56:59.727195 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:57:04.363750 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:04.363712 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg"] Jul 10 18:57:04.364176 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:04.363945 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" podUID="23ecad98-b686-4f6c-a045-13ff5fedd39f" containerName="istio-proxy" containerID="cri-o://846c1a4a1038e65e761bdcde112e61a95f013c4be305cf7f651390c392a823fe" gracePeriod=30 Jul 10 18:57:10.436942 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.436897 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n"] Jul 10 18:57:10.441701 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.441677 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.444173 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.444141 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b016bb74-openshift-default-dockercfg-92zt4\"" Jul 10 18:57:10.453422 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.453389 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n"] Jul 10 18:57:10.527334 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.527312 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:57:10.535787 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.535715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-data\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.535787 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.535755 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-podinfo\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.535954 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.535810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766ps\" (UniqueName: \"kubernetes.io/projected/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-kube-api-access-766ps\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.535954 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.535850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-workload-socket\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.535954 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.535888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istiod-ca-cert\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.535954 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.535913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-token\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.536159 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.535990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-workload-certs\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.536159 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.536028 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-credential-socket\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.536159 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.536048 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-envoy\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.637251 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637212 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fksr\" (UniqueName: \"kubernetes.io/projected/23ecad98-b686-4f6c-a045-13ff5fedd39f-kube-api-access-7fksr\") pod \"23ecad98-b686-4f6c-a045-13ff5fedd39f\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " Jul 10 18:57:10.637454 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637258 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-envoy\") pod \"23ecad98-b686-4f6c-a045-13ff5fedd39f\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " Jul 10 18:57:10.637454 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637325 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-workload-socket\") pod \"23ecad98-b686-4f6c-a045-13ff5fedd39f\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " Jul 10 18:57:10.637454 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637370 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-workload-certs\") pod \"23ecad98-b686-4f6c-a045-13ff5fedd39f\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " Jul 10 18:57:10.637454 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637402 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-data\") pod \"23ecad98-b686-4f6c-a045-13ff5fedd39f\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " Jul 10 18:57:10.637663 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637456 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ecad98-b686-4f6c-a045-13ff5fedd39f-istiod-ca-cert\") pod \"23ecad98-b686-4f6c-a045-13ff5fedd39f\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " Jul 10 18:57:10.637663 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637502 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-podinfo\") pod \"23ecad98-b686-4f6c-a045-13ff5fedd39f\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " Jul 10 18:57:10.637663 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637534 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-token\") pod \"23ecad98-b686-4f6c-a045-13ff5fedd39f\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " Jul 10 18:57:10.637663 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637581 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-credential-socket\") pod \"23ecad98-b686-4f6c-a045-13ff5fedd39f\" (UID: \"23ecad98-b686-4f6c-a045-13ff5fedd39f\") " Jul 10 18:57:10.637663 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637586 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "23ecad98-b686-4f6c-a045-13ff5fedd39f" (UID: "23ecad98-b686-4f6c-a045-13ff5fedd39f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:57:10.637905 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-workload-socket\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.637905 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istiod-ca-cert\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.637905 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-token\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.637905 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637836 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-workload-certs\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.637905 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637866 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ecad98-b686-4f6c-a045-13ff5fedd39f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "23ecad98-b686-4f6c-a045-13ff5fedd39f" (UID: "23ecad98-b686-4f6c-a045-13ff5fedd39f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:57:10.637905 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637887 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-credential-socket\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.638207 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-envoy\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.638207 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-data\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.638207 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.637983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-podinfo\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.638207 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.638016 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "23ecad98-b686-4f6c-a045-13ff5fedd39f" (UID: "23ecad98-b686-4f6c-a045-13ff5fedd39f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:57:10.638207 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.638056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766ps\" (UniqueName: \"kubernetes.io/projected/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-kube-api-access-766ps\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.638207 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.638098 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ecad98-b686-4f6c-a045-13ff5fedd39f-istiod-ca-cert\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:10.638207 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.638116 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-workload-socket\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:10.638207 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.638129 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-workload-certs\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:10.638635 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.638613 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "23ecad98-b686-4f6c-a045-13ff5fedd39f" (UID: "23ecad98-b686-4f6c-a045-13ff5fedd39f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:57:10.638950 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.638832 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-data" (OuterVolumeSpecName: "istio-data") pod "23ecad98-b686-4f6c-a045-13ff5fedd39f" (UID: "23ecad98-b686-4f6c-a045-13ff5fedd39f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:57:10.639070 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.638973 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-workload-socket\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.639740 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.639712 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-credential-socket\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.639866 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.639844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-data\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.640616 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.640230 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-workload-certs\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.640616 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.640430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istiod-ca-cert\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.641298 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.641262 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ecad98-b686-4f6c-a045-13ff5fedd39f-kube-api-access-7fksr" (OuterVolumeSpecName: "kube-api-access-7fksr") pod "23ecad98-b686-4f6c-a045-13ff5fedd39f" (UID: "23ecad98-b686-4f6c-a045-13ff5fedd39f"). InnerVolumeSpecName "kube-api-access-7fksr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:57:10.641634 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.641598 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-token" (OuterVolumeSpecName: "istio-token") pod "23ecad98-b686-4f6c-a045-13ff5fedd39f" (UID: "23ecad98-b686-4f6c-a045-13ff5fedd39f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:57:10.642132 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.642108 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "23ecad98-b686-4f6c-a045-13ff5fedd39f" (UID: "23ecad98-b686-4f6c-a045-13ff5fedd39f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:57:10.642320 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.642292 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "23ecad98-b686-4f6c-a045-13ff5fedd39f" (UID: "23ecad98-b686-4f6c-a045-13ff5fedd39f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 18:57:10.642693 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.642677 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-envoy\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.643092 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.643075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-podinfo\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.646749 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.646723 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-766ps\" (UniqueName: \"kubernetes.io/projected/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-kube-api-access-766ps\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.646919 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.646901 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-token\") pod \"e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.739446 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.739406 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-credential-socket\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:10.739446 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.739435 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fksr\" (UniqueName: \"kubernetes.io/projected/23ecad98-b686-4f6c-a045-13ff5fedd39f-kube-api-access-7fksr\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:10.739446 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.739445 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-envoy\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:10.739446 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.739453 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-data\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:10.739446 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.739460 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-podinfo\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:10.739842 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.739468 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ecad98-b686-4f6c-a045-13ff5fedd39f-istio-token\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:10.754297 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.754272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:10.793549 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.793473 2576 generic.go:356] "Generic (PLEG): container finished" podID="23ecad98-b686-4f6c-a045-13ff5fedd39f" containerID="846c1a4a1038e65e761bdcde112e61a95f013c4be305cf7f651390c392a823fe" exitCode=0 Jul 10 18:57:10.793549 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.793513 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" event={"ID":"23ecad98-b686-4f6c-a045-13ff5fedd39f","Type":"ContainerDied","Data":"846c1a4a1038e65e761bdcde112e61a95f013c4be305cf7f651390c392a823fe"} Jul 10 18:57:10.793549 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.793517 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" Jul 10 18:57:10.793758 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.793562 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg" event={"ID":"23ecad98-b686-4f6c-a045-13ff5fedd39f","Type":"ContainerDied","Data":"ee2668e32211b12bf94536c008ec886bafa55f08159b3bcba2e7b7777548dc22"} Jul 10 18:57:10.793758 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.793585 2576 scope.go:117] "RemoveContainer" containerID="846c1a4a1038e65e761bdcde112e61a95f013c4be305cf7f651390c392a823fe" Jul 10 18:57:10.804236 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.804213 2576 scope.go:117] "RemoveContainer" containerID="846c1a4a1038e65e761bdcde112e61a95f013c4be305cf7f651390c392a823fe" Jul 10 18:57:10.804541 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:57:10.804519 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846c1a4a1038e65e761bdcde112e61a95f013c4be305cf7f651390c392a823fe\": container with ID starting with 846c1a4a1038e65e761bdcde112e61a95f013c4be305cf7f651390c392a823fe not found: ID does not exist" containerID="846c1a4a1038e65e761bdcde112e61a95f013c4be305cf7f651390c392a823fe" Jul 10 18:57:10.804595 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.804556 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846c1a4a1038e65e761bdcde112e61a95f013c4be305cf7f651390c392a823fe"} err="failed to get container status \"846c1a4a1038e65e761bdcde112e61a95f013c4be305cf7f651390c392a823fe\": rpc error: code = NotFound desc = could not find container \"846c1a4a1038e65e761bdcde112e61a95f013c4be305cf7f651390c392a823fe\": container with ID starting with 846c1a4a1038e65e761bdcde112e61a95f013c4be305cf7f651390c392a823fe not found: ID does not exist" Jul 10 18:57:10.821168 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.821139 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg"] Jul 10 18:57:10.824809 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.824778 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d487a3ed-openshift-default-867754fdb7-fkzsg"] Jul 10 18:57:10.899831 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.899804 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n"] Jul 10 18:57:10.901114 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:57:10.901087 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4440e9b3_bf3b_4a4c_af44_9a3d7fe14749.slice/crio-d5f2560164eb5c10c523739440a70d9826f6a10dc8175e7f577197ec9f0144e2 WatchSource:0}: Error finding container d5f2560164eb5c10c523739440a70d9826f6a10dc8175e7f577197ec9f0144e2: Status 404 returned error can't find the container with id d5f2560164eb5c10c523739440a70d9826f6a10dc8175e7f577197ec9f0144e2 Jul 10 18:57:10.903136 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.903102 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:57:10.903244 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.903169 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:57:10.903244 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:10.903199 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:57:11.801189 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:11.801144 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" event={"ID":"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749","Type":"ContainerStarted","Data":"64d3408e517938fe93f1000cd07a3804dda6c3b1b6c140eb35df4e7ba7d7c2e8"} Jul 10 18:57:11.801624 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:11.801208 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" event={"ID":"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749","Type":"ContainerStarted","Data":"d5f2560164eb5c10c523739440a70d9826f6a10dc8175e7f577197ec9f0144e2"} Jul 10 18:57:11.842464 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:11.842413 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" podStartSLOduration=1.842397055 podStartE2EDuration="1.842397055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:57:11.841411205 +0000 UTC m=+1678.519123668" watchObservedRunningTime="2026-07-10 18:57:11.842397055 +0000 UTC m=+1678.520109517" Jul 10 18:57:11.845646 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:11.845622 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23ecad98-b686-4f6c-a045-13ff5fedd39f" path="/var/lib/kubelet/pods/23ecad98-b686-4f6c-a045-13ff5fedd39f/volumes" Jul 10 18:57:12.754797 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:12.754761 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:12.759870 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:12.759847 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:12.808072 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:12.808040 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:12.809030 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:12.809011 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:15.189769 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:15.189734 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:57:23.334704 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.334669 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n"] Jul 10 18:57:23.335166 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.334902 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" podUID="4440e9b3-bf3b-4a4c-af44-9a3d7fe14749" containerName="istio-proxy" containerID="cri-o://64d3408e517938fe93f1000cd07a3804dda6c3b1b6c140eb35df4e7ba7d7c2e8" gracePeriod=30 Jul 10 18:57:23.479743 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.479720 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:23.560183 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.560158 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-766ps\" (UniqueName: \"kubernetes.io/projected/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-kube-api-access-766ps\") pod \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " Jul 10 18:57:23.560304 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.560213 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-workload-socket\") pod \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " Jul 10 18:57:23.560304 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.560242 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-podinfo\") pod \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " Jul 10 18:57:23.560304 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.560273 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-workload-certs\") pod \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " Jul 10 18:57:23.560505 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.560305 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-data\") pod \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " Jul 10 18:57:23.560505 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.560378 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istiod-ca-cert\") pod \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " Jul 10 18:57:23.560505 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.560413 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-envoy\") pod \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " Jul 10 18:57:23.560505 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.560450 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-token\") pod \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " Jul 10 18:57:23.560505 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.560473 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-credential-socket\") pod \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\" (UID: \"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749\") " Jul 10 18:57:23.560774 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.560597 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749" (UID: "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:57:23.560774 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.560613 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749" (UID: "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:57:23.560884 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.560779 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749" (UID: "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:57:23.560884 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.560849 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-workload-socket\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:23.560884 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.560870 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-workload-certs\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:23.561042 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.560884 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-credential-socket\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:23.561228 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.561203 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749" (UID: "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:57:23.561305 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.561216 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-data" (OuterVolumeSpecName: "istio-data") pod "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749" (UID: "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:57:23.562516 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.562495 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749" (UID: "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 18:57:23.562516 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.562488 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-kube-api-access-766ps" (OuterVolumeSpecName: "kube-api-access-766ps") pod "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749" (UID: "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749"). InnerVolumeSpecName "kube-api-access-766ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:57:23.562892 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.562870 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749" (UID: "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:57:23.562958 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.562876 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-token" (OuterVolumeSpecName: "istio-token") pod "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749" (UID: "4440e9b3-bf3b-4a4c-af44-9a3d7fe14749"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:57:23.662249 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.662161 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-podinfo\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:23.662249 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.662187 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-data\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:23.662249 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.662196 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istiod-ca-cert\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:23.662249 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.662205 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-envoy\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:23.662249 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.662212 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-istio-token\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:23.662249 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.662220 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-766ps\" (UniqueName: \"kubernetes.io/projected/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749-kube-api-access-766ps\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:23.883196 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.883159 2576 generic.go:356] "Generic (PLEG): container finished" podID="4440e9b3-bf3b-4a4c-af44-9a3d7fe14749" containerID="64d3408e517938fe93f1000cd07a3804dda6c3b1b6c140eb35df4e7ba7d7c2e8" exitCode=0 Jul 10 18:57:23.883384 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.883209 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" Jul 10 18:57:23.883384 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.883245 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" event={"ID":"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749","Type":"ContainerDied","Data":"64d3408e517938fe93f1000cd07a3804dda6c3b1b6c140eb35df4e7ba7d7c2e8"} Jul 10 18:57:23.883384 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.883309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n" event={"ID":"4440e9b3-bf3b-4a4c-af44-9a3d7fe14749","Type":"ContainerDied","Data":"d5f2560164eb5c10c523739440a70d9826f6a10dc8175e7f577197ec9f0144e2"} Jul 10 18:57:23.883384 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.883340 2576 scope.go:117] "RemoveContainer" containerID="64d3408e517938fe93f1000cd07a3804dda6c3b1b6c140eb35df4e7ba7d7c2e8" Jul 10 18:57:23.892162 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.892144 2576 scope.go:117] "RemoveContainer" containerID="64d3408e517938fe93f1000cd07a3804dda6c3b1b6c140eb35df4e7ba7d7c2e8" Jul 10 18:57:23.892428 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:57:23.892408 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d3408e517938fe93f1000cd07a3804dda6c3b1b6c140eb35df4e7ba7d7c2e8\": container with ID starting with 64d3408e517938fe93f1000cd07a3804dda6c3b1b6c140eb35df4e7ba7d7c2e8 not found: ID does not exist" containerID="64d3408e517938fe93f1000cd07a3804dda6c3b1b6c140eb35df4e7ba7d7c2e8" Jul 10 18:57:23.892483 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.892436 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d3408e517938fe93f1000cd07a3804dda6c3b1b6c140eb35df4e7ba7d7c2e8"} err="failed to get container status \"64d3408e517938fe93f1000cd07a3804dda6c3b1b6c140eb35df4e7ba7d7c2e8\": rpc error: code = NotFound desc = could not find container \"64d3408e517938fe93f1000cd07a3804dda6c3b1b6c140eb35df4e7ba7d7c2e8\": container with ID starting with 64d3408e517938fe93f1000cd07a3804dda6c3b1b6c140eb35df4e7ba7d7c2e8 not found: ID does not exist" Jul 10 18:57:23.901369 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.901337 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n"] Jul 10 18:57:23.905505 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:23.905486 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b016bb74-openshift-default-75c676b89f-rsn2n"] Jul 10 18:57:25.843521 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:25.843484 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4440e9b3-bf3b-4a4c-af44-9a3d7fe14749" path="/var/lib/kubelet/pods/4440e9b3-bf3b-4a4c-af44-9a3d7fe14749/volumes" Jul 10 18:57:27.486767 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:27.486731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:57:29.498491 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.498461 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4"] Jul 10 18:57:29.498896 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.498795 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ecad98-b686-4f6c-a045-13ff5fedd39f" containerName="istio-proxy" Jul 10 18:57:29.498896 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.498807 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ecad98-b686-4f6c-a045-13ff5fedd39f" containerName="istio-proxy" Jul 10 18:57:29.498896 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.498826 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4440e9b3-bf3b-4a4c-af44-9a3d7fe14749" containerName="istio-proxy" Jul 10 18:57:29.498896 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.498832 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4440e9b3-bf3b-4a4c-af44-9a3d7fe14749" containerName="istio-proxy" Jul 10 18:57:29.498896 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.498883 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ecad98-b686-4f6c-a045-13ff5fedd39f" containerName="istio-proxy" Jul 10 18:57:29.498896 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.498896 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4440e9b3-bf3b-4a4c-af44-9a3d7fe14749" containerName="istio-proxy" Jul 10 18:57:29.501397 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.501379 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.503822 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.503798 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-77dcb9ba-openshift-default-dockercfg-8wjqb\"" Jul 10 18:57:29.514332 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.514305 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4"] Jul 10 18:57:29.613773 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.613735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-envoy\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.613966 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.613820 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-data\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.613966 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.613861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istiod-ca-cert\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.614081 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.613960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-credential-socket\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.614081 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.613994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5vwg\" (UniqueName: \"kubernetes.io/projected/45416dcf-9e43-424f-95e8-5b6679d9ccc2-kube-api-access-l5vwg\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.614081 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.614052 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-workload-socket\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.614211 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.614088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-token\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.614211 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.614116 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-workload-certs\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.614211 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.614131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-podinfo\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.714516 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.714483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-data\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.714516 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.714522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istiod-ca-cert\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.714741 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.714564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-credential-socket\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.714741 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.714580 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5vwg\" (UniqueName: \"kubernetes.io/projected/45416dcf-9e43-424f-95e8-5b6679d9ccc2-kube-api-access-l5vwg\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.714741 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.714603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-workload-socket\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.714741 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.714622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-token\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.714741 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.714643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-workload-certs\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.714741 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.714687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-podinfo\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.715058 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.714746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-envoy\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.715058 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.714933 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-data\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.715058 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.714997 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-credential-socket\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.715217 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.715146 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-workload-socket\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.715217 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.715197 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-workload-certs\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.715460 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.715438 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istiod-ca-cert\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.717100 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.717080 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-envoy\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.717224 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.717207 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-podinfo\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.722773 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.722744 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5vwg\" (UniqueName: \"kubernetes.io/projected/45416dcf-9e43-424f-95e8-5b6679d9ccc2-kube-api-access-l5vwg\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.723058 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.723036 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-token\") pod \"e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.815307 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.815274 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:29.966466 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.966435 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4"] Jul 10 18:57:29.967697 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:57:29.967665 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45416dcf_9e43_424f_95e8_5b6679d9ccc2.slice/crio-7fcc98f1a4adb1c34572619867c1dc4353504fe9cfcb1efd5b9bc2a9e6a045fc WatchSource:0}: Error finding container 7fcc98f1a4adb1c34572619867c1dc4353504fe9cfcb1efd5b9bc2a9e6a045fc: Status 404 returned error can't find the container with id 7fcc98f1a4adb1c34572619867c1dc4353504fe9cfcb1efd5b9bc2a9e6a045fc Jul 10 18:57:29.969942 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.969914 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:57:29.970036 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.969973 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:57:29.970036 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:29.970001 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:57:30.923245 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:30.923205 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" event={"ID":"45416dcf-9e43-424f-95e8-5b6679d9ccc2","Type":"ContainerStarted","Data":"df6e2f188383882c50505431d5b1118851dd94a3e1bf9312eea5a13f04f6627f"} Jul 10 18:57:30.923655 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:30.923285 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" event={"ID":"45416dcf-9e43-424f-95e8-5b6679d9ccc2","Type":"ContainerStarted","Data":"7fcc98f1a4adb1c34572619867c1dc4353504fe9cfcb1efd5b9bc2a9e6a045fc"} Jul 10 18:57:30.964590 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:30.964536 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" podStartSLOduration=1.964519924 podStartE2EDuration="1.964519924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:57:30.963297192 +0000 UTC m=+1697.641009655" watchObservedRunningTime="2026-07-10 18:57:30.964519924 +0000 UTC m=+1697.642232387" Jul 10 18:57:31.816242 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:31.816206 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:31.821445 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:31.821419 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:31.928777 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:31.928744 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:31.929622 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:31.929597 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:34.187190 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:34.187152 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:57:35.206187 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:35.206156 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4"] Jul 10 18:57:35.206622 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:35.206327 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" podUID="45416dcf-9e43-424f-95e8-5b6679d9ccc2" containerName="istio-proxy" containerID="cri-o://df6e2f188383882c50505431d5b1118851dd94a3e1bf9312eea5a13f04f6627f" gracePeriod=30 Jul 10 18:57:42.165058 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.165025 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:42.228997 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.228910 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5vwg\" (UniqueName: \"kubernetes.io/projected/45416dcf-9e43-424f-95e8-5b6679d9ccc2-kube-api-access-l5vwg\") pod \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " Jul 10 18:57:42.228997 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.228968 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istiod-ca-cert\") pod \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " Jul 10 18:57:42.229211 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.228992 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-envoy\") pod \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " Jul 10 18:57:42.229211 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.229017 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-workload-socket\") pod \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " Jul 10 18:57:42.229211 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.229039 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-credential-socket\") pod \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " Jul 10 18:57:42.229211 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.229067 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-podinfo\") pod \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " Jul 10 18:57:42.229211 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.229091 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-token\") pod \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " Jul 10 18:57:42.229211 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.229141 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-data\") pod \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " Jul 10 18:57:42.229211 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.229164 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-workload-certs\") pod \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\" (UID: \"45416dcf-9e43-424f-95e8-5b6679d9ccc2\") " Jul 10 18:57:42.229605 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.229380 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45416dcf-9e43-424f-95e8-5b6679d9ccc2" (UID: "45416dcf-9e43-424f-95e8-5b6679d9ccc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:57:42.229605 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.229430 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45416dcf-9e43-424f-95e8-5b6679d9ccc2" (UID: "45416dcf-9e43-424f-95e8-5b6679d9ccc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:57:42.229715 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.229611 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-workload-socket\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:42.229715 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.229649 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-credential-socket\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:42.229826 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.229775 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45416dcf-9e43-424f-95e8-5b6679d9ccc2" (UID: "45416dcf-9e43-424f-95e8-5b6679d9ccc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:57:42.230099 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.230077 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45416dcf-9e43-424f-95e8-5b6679d9ccc2" (UID: "45416dcf-9e43-424f-95e8-5b6679d9ccc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:57:42.230374 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.230248 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-data" (OuterVolumeSpecName: "istio-data") pod "45416dcf-9e43-424f-95e8-5b6679d9ccc2" (UID: "45416dcf-9e43-424f-95e8-5b6679d9ccc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:57:42.231674 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.231648 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-token" (OuterVolumeSpecName: "istio-token") pod "45416dcf-9e43-424f-95e8-5b6679d9ccc2" (UID: "45416dcf-9e43-424f-95e8-5b6679d9ccc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:57:42.231772 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.231649 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45416dcf-9e43-424f-95e8-5b6679d9ccc2" (UID: "45416dcf-9e43-424f-95e8-5b6679d9ccc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:57:42.231859 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.231837 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45416dcf-9e43-424f-95e8-5b6679d9ccc2-kube-api-access-l5vwg" (OuterVolumeSpecName: "kube-api-access-l5vwg") pod "45416dcf-9e43-424f-95e8-5b6679d9ccc2" (UID: "45416dcf-9e43-424f-95e8-5b6679d9ccc2"). InnerVolumeSpecName "kube-api-access-l5vwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:57:42.232030 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.232004 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45416dcf-9e43-424f-95e8-5b6679d9ccc2" (UID: "45416dcf-9e43-424f-95e8-5b6679d9ccc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 18:57:42.330504 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.330470 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5vwg\" (UniqueName: \"kubernetes.io/projected/45416dcf-9e43-424f-95e8-5b6679d9ccc2-kube-api-access-l5vwg\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:42.330504 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.330498 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istiod-ca-cert\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:42.330504 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.330507 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-envoy\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:42.330723 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.330515 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-podinfo\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:42.330723 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.330522 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-token\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:42.330723 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.330530 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-istio-data\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:42.330723 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.330537 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45416dcf-9e43-424f-95e8-5b6679d9ccc2-workload-certs\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:57:42.992430 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.992399 2576 generic.go:356] "Generic (PLEG): container finished" podID="45416dcf-9e43-424f-95e8-5b6679d9ccc2" containerID="df6e2f188383882c50505431d5b1118851dd94a3e1bf9312eea5a13f04f6627f" exitCode=0 Jul 10 18:57:42.992646 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.992453 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" event={"ID":"45416dcf-9e43-424f-95e8-5b6679d9ccc2","Type":"ContainerDied","Data":"df6e2f188383882c50505431d5b1118851dd94a3e1bf9312eea5a13f04f6627f"} Jul 10 18:57:42.992646 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.992484 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" Jul 10 18:57:42.992646 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.992498 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4" event={"ID":"45416dcf-9e43-424f-95e8-5b6679d9ccc2","Type":"ContainerDied","Data":"7fcc98f1a4adb1c34572619867c1dc4353504fe9cfcb1efd5b9bc2a9e6a045fc"} Jul 10 18:57:42.992646 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:42.992518 2576 scope.go:117] "RemoveContainer" containerID="df6e2f188383882c50505431d5b1118851dd94a3e1bf9312eea5a13f04f6627f" Jul 10 18:57:43.001802 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:43.001781 2576 scope.go:117] "RemoveContainer" containerID="df6e2f188383882c50505431d5b1118851dd94a3e1bf9312eea5a13f04f6627f" Jul 10 18:57:43.002113 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:57:43.002092 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df6e2f188383882c50505431d5b1118851dd94a3e1bf9312eea5a13f04f6627f\": container with ID starting with df6e2f188383882c50505431d5b1118851dd94a3e1bf9312eea5a13f04f6627f not found: ID does not exist" containerID="df6e2f188383882c50505431d5b1118851dd94a3e1bf9312eea5a13f04f6627f" Jul 10 18:57:43.002197 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:43.002125 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df6e2f188383882c50505431d5b1118851dd94a3e1bf9312eea5a13f04f6627f"} err="failed to get container status \"df6e2f188383882c50505431d5b1118851dd94a3e1bf9312eea5a13f04f6627f\": rpc error: code = NotFound desc = could not find container \"df6e2f188383882c50505431d5b1118851dd94a3e1bf9312eea5a13f04f6627f\": container with ID starting with df6e2f188383882c50505431d5b1118851dd94a3e1bf9312eea5a13f04f6627f not found: ID does not exist" Jul 10 18:57:43.015380 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:43.015341 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4"] Jul 10 18:57:43.021848 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:43.021827 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-77dcb9ba-openshift-default-74cf55ff87-88pb4"] Jul 10 18:57:43.843782 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:43.843750 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45416dcf-9e43-424f-95e8-5b6679d9ccc2" path="/var/lib/kubelet/pods/45416dcf-9e43-424f-95e8-5b6679d9ccc2/volumes" Jul 10 18:57:45.900537 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:57:45.900506 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:59:04.190382 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:04.190334 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:59:13.090213 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:13.090179 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:59:13.910601 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:13.910568 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:59:13.914567 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:13.914545 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 18:59:15.422446 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.422417 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d"] Jul 10 18:59:15.422811 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.422797 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45416dcf-9e43-424f-95e8-5b6679d9ccc2" containerName="istio-proxy" Jul 10 18:59:15.422811 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.422809 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="45416dcf-9e43-424f-95e8-5b6679d9ccc2" containerName="istio-proxy" Jul 10 18:59:15.422888 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.422880 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="45416dcf-9e43-424f-95e8-5b6679d9ccc2" containerName="istio-proxy" Jul 10 18:59:15.425808 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.425791 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.428726 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.428707 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c9361313-openshift-default-dockercfg-fl9ph\"" Jul 10 18:59:15.452227 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.452202 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d"] Jul 10 18:59:15.478534 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.478508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-envoy\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.478679 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.478544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e188edff-1ab2-43d4-adf9-86077f48dd42-istiod-ca-cert\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.478679 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.478572 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9qhv\" (UniqueName: \"kubernetes.io/projected/e188edff-1ab2-43d4-adf9-86077f48dd42-kube-api-access-k9qhv\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.478679 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.478622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-token\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.478679 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.478650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-credential-socket\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.478871 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.478681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-workload-socket\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.478871 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.478710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-data\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.478871 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.478728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-podinfo\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.478871 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.478774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-workload-certs\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.579991 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.579955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-envoy\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.579991 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.579996 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e188edff-1ab2-43d4-adf9-86077f48dd42-istiod-ca-cert\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.580219 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.580023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9qhv\" (UniqueName: \"kubernetes.io/projected/e188edff-1ab2-43d4-adf9-86077f48dd42-kube-api-access-k9qhv\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.580219 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.580135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-token\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.580219 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.580191 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-credential-socket\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.580417 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.580234 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-workload-socket\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.580417 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.580278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-data\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.580417 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.580306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-podinfo\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.580417 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.580342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-workload-certs\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.580642 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.580622 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-workload-socket\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.580698 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.580656 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-data\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.580759 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.580713 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-workload-certs\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.580825 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.580809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e188edff-1ab2-43d4-adf9-86077f48dd42-istiod-ca-cert\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.580939 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.580918 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-credential-socket\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.582421 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.582400 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-envoy\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.582712 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.582695 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-podinfo\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.587988 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.587963 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9qhv\" (UniqueName: \"kubernetes.io/projected/e188edff-1ab2-43d4-adf9-86077f48dd42-kube-api-access-k9qhv\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.588154 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.588138 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-token\") pod \"e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.737487 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.737398 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:15.884151 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.884118 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d"] Jul 10 18:59:15.886516 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:59:15.886487 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode188edff_1ab2_43d4_adf9_86077f48dd42.slice/crio-0b43241692e350fa021675381693e6b32fc9dced4bba9f53f9362759e1d02e99 WatchSource:0}: Error finding container 0b43241692e350fa021675381693e6b32fc9dced4bba9f53f9362759e1d02e99: Status 404 returned error can't find the container with id 0b43241692e350fa021675381693e6b32fc9dced4bba9f53f9362759e1d02e99 Jul 10 18:59:15.888523 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.888489 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:59:15.888674 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.888572 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:59:15.888674 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:15.888626 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:59:16.479019 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:16.478986 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" event={"ID":"e188edff-1ab2-43d4-adf9-86077f48dd42","Type":"ContainerStarted","Data":"fb5fceb8b0feb9acdcab04e73dd65338284dbd9e0ae808ed39160d7a77672a6b"} Jul 10 18:59:16.479428 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:16.479049 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" event={"ID":"e188edff-1ab2-43d4-adf9-86077f48dd42","Type":"ContainerStarted","Data":"0b43241692e350fa021675381693e6b32fc9dced4bba9f53f9362759e1d02e99"} Jul 10 18:59:16.506323 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:16.506277 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" podStartSLOduration=1.506261114 podStartE2EDuration="1.506261114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:16.504782115 +0000 UTC m=+1803.182494577" watchObservedRunningTime="2026-07-10 18:59:16.506261114 +0000 UTC m=+1803.183973575" Jul 10 18:59:16.737963 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:16.737863 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:16.742813 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:16.742780 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:17.483854 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:17.483818 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:17.484709 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:17.484691 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:23.994379 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:23.994335 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:59:32.485728 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:32.485689 2576 patch_prober.go:28] interesting pod/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 10 18:59:32.486213 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:32.485766 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" podUID="e188edff-1ab2-43d4-adf9-86077f48dd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 10 18:59:32.574921 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:32.574890 2576 generic.go:356] "Generic (PLEG): container finished" podID="e188edff-1ab2-43d4-adf9-86077f48dd42" containerID="fb5fceb8b0feb9acdcab04e73dd65338284dbd9e0ae808ed39160d7a77672a6b" exitCode=0 Jul 10 18:59:32.575088 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:32.574965 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" event={"ID":"e188edff-1ab2-43d4-adf9-86077f48dd42","Type":"ContainerDied","Data":"fb5fceb8b0feb9acdcab04e73dd65338284dbd9e0ae808ed39160d7a77672a6b"} Jul 10 18:59:32.575341 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:32.575324 2576 scope.go:117] "RemoveContainer" containerID="fb5fceb8b0feb9acdcab04e73dd65338284dbd9e0ae808ed39160d7a77672a6b" Jul 10 18:59:32.576378 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:32.576334 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:59:32.576482 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:32.576421 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:59:32.576482 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:32.576458 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:59:33.582311 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:33.582278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" event={"ID":"e188edff-1ab2-43d4-adf9-86077f48dd42","Type":"ContainerStarted","Data":"c88ca7c342412da1ac53fbb95410251b90a7e2b90024d3967eb7e0b45232d515"} Jul 10 18:59:33.738178 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:33.738138 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:33.738399 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:33.738191 2576 patch_prober.go:28] interesting pod/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 10 18:59:33.738399 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:33.738257 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" podUID="e188edff-1ab2-43d4-adf9-86077f48dd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 10 18:59:34.738301 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:34.738263 2576 patch_prober.go:28] interesting pod/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 10 18:59:34.738914 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:34.738880 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" podUID="e188edff-1ab2-43d4-adf9-86077f48dd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 10 18:59:35.120303 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:35.120267 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 18:59:35.737995 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:35.737957 2576 patch_prober.go:28] interesting pod/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 10 18:59:35.738282 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:35.738250 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" podUID="e188edff-1ab2-43d4-adf9-86077f48dd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 10 18:59:36.596601 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:36.596564 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d"] Jul 10 18:59:36.597033 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:36.596805 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" podUID="e188edff-1ab2-43d4-adf9-86077f48dd42" containerName="istio-proxy" containerID="cri-o://c88ca7c342412da1ac53fbb95410251b90a7e2b90024d3967eb7e0b45232d515" gracePeriod=30 Jul 10 18:59:42.550552 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.550528 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:42.630707 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.630626 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-credential-socket\") pod \"e188edff-1ab2-43d4-adf9-86077f48dd42\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " Jul 10 18:59:42.630707 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.630670 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-workload-socket\") pod \"e188edff-1ab2-43d4-adf9-86077f48dd42\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " Jul 10 18:59:42.630925 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.630740 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-envoy\") pod \"e188edff-1ab2-43d4-adf9-86077f48dd42\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " Jul 10 18:59:42.630925 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.630775 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9qhv\" (UniqueName: \"kubernetes.io/projected/e188edff-1ab2-43d4-adf9-86077f48dd42-kube-api-access-k9qhv\") pod \"e188edff-1ab2-43d4-adf9-86077f48dd42\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " Jul 10 18:59:42.630925 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.630797 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-workload-certs\") pod \"e188edff-1ab2-43d4-adf9-86077f48dd42\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " Jul 10 18:59:42.631089 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.630945 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e188edff-1ab2-43d4-adf9-86077f48dd42" (UID: "e188edff-1ab2-43d4-adf9-86077f48dd42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:59:42.631089 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.631022 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-token\") pod \"e188edff-1ab2-43d4-adf9-86077f48dd42\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " Jul 10 18:59:42.631089 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.631064 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-podinfo\") pod \"e188edff-1ab2-43d4-adf9-86077f48dd42\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " Jul 10 18:59:42.631089 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.631066 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e188edff-1ab2-43d4-adf9-86077f48dd42" (UID: "e188edff-1ab2-43d4-adf9-86077f48dd42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:59:42.631089 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.631077 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e188edff-1ab2-43d4-adf9-86077f48dd42" (UID: "e188edff-1ab2-43d4-adf9-86077f48dd42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:59:42.631370 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.631106 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e188edff-1ab2-43d4-adf9-86077f48dd42-istiod-ca-cert\") pod \"e188edff-1ab2-43d4-adf9-86077f48dd42\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " Jul 10 18:59:42.631370 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.631145 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-data\") pod \"e188edff-1ab2-43d4-adf9-86077f48dd42\" (UID: \"e188edff-1ab2-43d4-adf9-86077f48dd42\") " Jul 10 18:59:42.631519 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.631498 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-credential-socket\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:59:42.631575 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.631526 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-workload-socket\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:59:42.631575 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.631524 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e188edff-1ab2-43d4-adf9-86077f48dd42-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e188edff-1ab2-43d4-adf9-86077f48dd42" (UID: "e188edff-1ab2-43d4-adf9-86077f48dd42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:59:42.631575 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.631541 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-workload-certs\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:59:42.632564 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.632534 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-data" (OuterVolumeSpecName: "istio-data") pod "e188edff-1ab2-43d4-adf9-86077f48dd42" (UID: "e188edff-1ab2-43d4-adf9-86077f48dd42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:59:42.633546 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.633513 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e188edff-1ab2-43d4-adf9-86077f48dd42" (UID: "e188edff-1ab2-43d4-adf9-86077f48dd42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 18:59:42.633750 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.633727 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-token" (OuterVolumeSpecName: "istio-token") pod "e188edff-1ab2-43d4-adf9-86077f48dd42" (UID: "e188edff-1ab2-43d4-adf9-86077f48dd42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:59:42.633875 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.633842 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e188edff-1ab2-43d4-adf9-86077f48dd42-kube-api-access-k9qhv" (OuterVolumeSpecName: "kube-api-access-k9qhv") pod "e188edff-1ab2-43d4-adf9-86077f48dd42" (UID: "e188edff-1ab2-43d4-adf9-86077f48dd42"). InnerVolumeSpecName "kube-api-access-k9qhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:59:42.633987 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.633960 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e188edff-1ab2-43d4-adf9-86077f48dd42" (UID: "e188edff-1ab2-43d4-adf9-86077f48dd42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:59:42.636803 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.636778 2576 generic.go:356] "Generic (PLEG): container finished" podID="e188edff-1ab2-43d4-adf9-86077f48dd42" containerID="c88ca7c342412da1ac53fbb95410251b90a7e2b90024d3967eb7e0b45232d515" exitCode=0 Jul 10 18:59:42.636898 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.636853 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" Jul 10 18:59:42.636898 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.636869 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" event={"ID":"e188edff-1ab2-43d4-adf9-86077f48dd42","Type":"ContainerDied","Data":"c88ca7c342412da1ac53fbb95410251b90a7e2b90024d3967eb7e0b45232d515"} Jul 10 18:59:42.637026 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.636939 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d" event={"ID":"e188edff-1ab2-43d4-adf9-86077f48dd42","Type":"ContainerDied","Data":"0b43241692e350fa021675381693e6b32fc9dced4bba9f53f9362759e1d02e99"} Jul 10 18:59:42.637026 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.636972 2576 scope.go:117] "RemoveContainer" containerID="c88ca7c342412da1ac53fbb95410251b90a7e2b90024d3967eb7e0b45232d515" Jul 10 18:59:42.653097 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.653077 2576 scope.go:117] "RemoveContainer" containerID="fb5fceb8b0feb9acdcab04e73dd65338284dbd9e0ae808ed39160d7a77672a6b" Jul 10 18:59:42.662011 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.661993 2576 scope.go:117] "RemoveContainer" containerID="c88ca7c342412da1ac53fbb95410251b90a7e2b90024d3967eb7e0b45232d515" Jul 10 18:59:42.662311 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:59:42.662291 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88ca7c342412da1ac53fbb95410251b90a7e2b90024d3967eb7e0b45232d515\": container with ID starting with c88ca7c342412da1ac53fbb95410251b90a7e2b90024d3967eb7e0b45232d515 not found: ID does not exist" containerID="c88ca7c342412da1ac53fbb95410251b90a7e2b90024d3967eb7e0b45232d515" Jul 10 18:59:42.662405 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.662319 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88ca7c342412da1ac53fbb95410251b90a7e2b90024d3967eb7e0b45232d515"} err="failed to get container status \"c88ca7c342412da1ac53fbb95410251b90a7e2b90024d3967eb7e0b45232d515\": rpc error: code = NotFound desc = could not find container \"c88ca7c342412da1ac53fbb95410251b90a7e2b90024d3967eb7e0b45232d515\": container with ID starting with c88ca7c342412da1ac53fbb95410251b90a7e2b90024d3967eb7e0b45232d515 not found: ID does not exist" Jul 10 18:59:42.662405 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.662339 2576 scope.go:117] "RemoveContainer" containerID="fb5fceb8b0feb9acdcab04e73dd65338284dbd9e0ae808ed39160d7a77672a6b" Jul 10 18:59:42.662603 ip-10-0-140-180 kubenswrapper[2576]: E0710 18:59:42.662586 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb5fceb8b0feb9acdcab04e73dd65338284dbd9e0ae808ed39160d7a77672a6b\": container with ID starting with fb5fceb8b0feb9acdcab04e73dd65338284dbd9e0ae808ed39160d7a77672a6b not found: ID does not exist" containerID="fb5fceb8b0feb9acdcab04e73dd65338284dbd9e0ae808ed39160d7a77672a6b" Jul 10 18:59:42.662649 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.662607 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5fceb8b0feb9acdcab04e73dd65338284dbd9e0ae808ed39160d7a77672a6b"} err="failed to get container status \"fb5fceb8b0feb9acdcab04e73dd65338284dbd9e0ae808ed39160d7a77672a6b\": rpc error: code = NotFound desc = could not find container \"fb5fceb8b0feb9acdcab04e73dd65338284dbd9e0ae808ed39160d7a77672a6b\": container with ID starting with fb5fceb8b0feb9acdcab04e73dd65338284dbd9e0ae808ed39160d7a77672a6b not found: ID does not exist" Jul 10 18:59:42.663642 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.663619 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d"] Jul 10 18:59:42.668142 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.668118 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c9361313-openshift-default-778d5f6c5f-dmh8d"] Jul 10 18:59:42.732407 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.732377 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-envoy\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:59:42.732407 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.732401 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9qhv\" (UniqueName: \"kubernetes.io/projected/e188edff-1ab2-43d4-adf9-86077f48dd42-kube-api-access-k9qhv\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:59:42.732407 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.732411 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-token\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:59:42.732644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.732419 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-podinfo\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:59:42.732644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.732426 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e188edff-1ab2-43d4-adf9-86077f48dd42-istiod-ca-cert\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:59:42.732644 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:42.732435 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e188edff-1ab2-43d4-adf9-86077f48dd42-istio-data\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 18:59:43.843114 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:43.843080 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e188edff-1ab2-43d4-adf9-86077f48dd42" path="/var/lib/kubelet/pods/e188edff-1ab2-43d4-adf9-86077f48dd42/volumes" Jul 10 18:59:56.293468 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.293431 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4"] Jul 10 18:59:56.293987 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.293898 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e188edff-1ab2-43d4-adf9-86077f48dd42" containerName="istio-proxy" Jul 10 18:59:56.293987 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.293918 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e188edff-1ab2-43d4-adf9-86077f48dd42" containerName="istio-proxy" Jul 10 18:59:56.293987 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.293949 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e188edff-1ab2-43d4-adf9-86077f48dd42" containerName="istio-proxy" Jul 10 18:59:56.293987 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.293958 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e188edff-1ab2-43d4-adf9-86077f48dd42" containerName="istio-proxy" Jul 10 18:59:56.294201 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.294026 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e188edff-1ab2-43d4-adf9-86077f48dd42" containerName="istio-proxy" Jul 10 18:59:56.294201 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.294035 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e188edff-1ab2-43d4-adf9-86077f48dd42" containerName="istio-proxy" Jul 10 18:59:56.297144 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.297129 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.299701 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.299678 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b39babdc-openshift-default-dockercfg-8jgrt\"" Jul 10 18:59:56.307450 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.307428 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4"] Jul 10 18:59:56.355314 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.355285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-credential-socket\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.355491 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.355317 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-istio-envoy\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.355491 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.355343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-workload-socket\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.355491 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.355436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-istio-data\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.355491 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.355484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1b9943-685d-49f6-8096-85d95d293155-istio-podinfo\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.355666 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.355519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1b9943-685d-49f6-8096-85d95d293155-istio-token\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.355666 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.355548 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-workload-certs\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.355666 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.355568 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1b9943-685d-49f6-8096-85d95d293155-istiod-ca-cert\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.355666 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.355588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fk66\" (UniqueName: \"kubernetes.io/projected/7e1b9943-685d-49f6-8096-85d95d293155-kube-api-access-4fk66\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.455974 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.455940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1b9943-685d-49f6-8096-85d95d293155-istio-podinfo\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.456161 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.455980 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1b9943-685d-49f6-8096-85d95d293155-istio-token\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.456161 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.456004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-workload-certs\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.456161 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.456024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1b9943-685d-49f6-8096-85d95d293155-istiod-ca-cert\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.456161 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.456040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fk66\" (UniqueName: \"kubernetes.io/projected/7e1b9943-685d-49f6-8096-85d95d293155-kube-api-access-4fk66\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.456161 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.456092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-credential-socket\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.456450 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.456224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-istio-envoy\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.456450 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.456276 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-workload-socket\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.456450 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.456330 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-istio-data\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.456606 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.456508 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-workload-certs\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.456606 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.456539 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-credential-socket\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.456695 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.456677 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-istio-data\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.456789 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.456765 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-workload-socket\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.457095 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.457074 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1b9943-685d-49f6-8096-85d95d293155-istiod-ca-cert\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.458344 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.458323 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-istio-envoy\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.458625 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.458604 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1b9943-685d-49f6-8096-85d95d293155-istio-podinfo\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.463686 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.463665 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1b9943-685d-49f6-8096-85d95d293155-istio-token\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.463976 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.463959 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fk66\" (UniqueName: \"kubernetes.io/projected/7e1b9943-685d-49f6-8096-85d95d293155-kube-api-access-4fk66\") pod \"e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.608452 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.608423 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:56.747265 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.747242 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4"] Jul 10 18:59:56.749688 ip-10-0-140-180 kubenswrapper[2576]: W0710 18:59:56.749654 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1b9943_685d_49f6_8096_85d95d293155.slice/crio-084d4bdeb911fc86046c9d8d42ede878bb4dcd90309efd79bee69581a7fceac5 WatchSource:0}: Error finding container 084d4bdeb911fc86046c9d8d42ede878bb4dcd90309efd79bee69581a7fceac5: Status 404 returned error can't find the container with id 084d4bdeb911fc86046c9d8d42ede878bb4dcd90309efd79bee69581a7fceac5 Jul 10 18:59:56.751703 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.751665 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:59:56.751815 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.751731 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:59:56.751815 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:56.751757 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 18:59:57.717721 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:57.717684 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" event={"ID":"7e1b9943-685d-49f6-8096-85d95d293155","Type":"ContainerStarted","Data":"7c5041d07cc625bd0e83f43ee58a587414a5e59ad9948f316b59f1a9980a216c"} Jul 10 18:59:57.718095 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:57.717740 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" event={"ID":"7e1b9943-685d-49f6-8096-85d95d293155","Type":"ContainerStarted","Data":"084d4bdeb911fc86046c9d8d42ede878bb4dcd90309efd79bee69581a7fceac5"} Jul 10 18:59:57.739941 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:57.739886 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" podStartSLOduration=1.739869383 podStartE2EDuration="1.739869383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:57.737550459 +0000 UTC m=+1844.415262923" watchObservedRunningTime="2026-07-10 18:59:57.739869383 +0000 UTC m=+1844.417581898" Jul 10 18:59:58.608555 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:58.608523 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:58.613713 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:58.613687 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:58.723296 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:58.723267 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 18:59:58.724106 ip-10-0-140-180 kubenswrapper[2576]: I0710 18:59:58.724089 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 19:00:05.488477 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:05.488440 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:00:13.725110 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:13.725035 2576 patch_prober.go:28] interesting pod/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 10 19:00:13.725110 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:13.725085 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" podUID="7e1b9943-685d-49f6-8096-85d95d293155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 10 19:00:13.805491 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:13.805406 2576 generic.go:356] "Generic (PLEG): container finished" podID="7e1b9943-685d-49f6-8096-85d95d293155" containerID="7c5041d07cc625bd0e83f43ee58a587414a5e59ad9948f316b59f1a9980a216c" exitCode=0 Jul 10 19:00:13.805491 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:13.805480 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" event={"ID":"7e1b9943-685d-49f6-8096-85d95d293155","Type":"ContainerDied","Data":"7c5041d07cc625bd0e83f43ee58a587414a5e59ad9948f316b59f1a9980a216c"} Jul 10 19:00:13.805876 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:13.805863 2576 scope.go:117] "RemoveContainer" containerID="7c5041d07cc625bd0e83f43ee58a587414a5e59ad9948f316b59f1a9980a216c" Jul 10 19:00:13.806824 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:13.806796 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 19:00:13.806925 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:13.806860 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 19:00:13.806925 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:13.806891 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 19:00:14.812851 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:14.812818 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" event={"ID":"7e1b9943-685d-49f6-8096-85d95d293155","Type":"ContainerStarted","Data":"ace421984988ba73f80c4a1b87cb5963b812000df6d00d45662d857567651d7a"} Jul 10 19:00:15.608732 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:15.608692 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 19:00:15.608935 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:15.608903 2576 patch_prober.go:28] interesting pod/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 10 19:00:15.609013 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:15.608962 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" podUID="7e1b9943-685d-49f6-8096-85d95d293155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 10 19:00:16.609371 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:16.609316 2576 patch_prober.go:28] interesting pod/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 10 19:00:16.609766 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:16.609393 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" podUID="7e1b9943-685d-49f6-8096-85d95d293155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 10 19:00:17.608830 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:17.608789 2576 patch_prober.go:28] interesting pod/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 10 19:00:17.609033 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:17.608848 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" podUID="7e1b9943-685d-49f6-8096-85d95d293155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 10 19:00:18.609093 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:18.609055 2576 patch_prober.go:28] interesting pod/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 10 19:00:18.609515 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:18.609110 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" podUID="7e1b9943-685d-49f6-8096-85d95d293155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 10 19:00:19.612833 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:19.612805 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 19:00:19.613211 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:19.613061 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 19:00:19.613650 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:19.613633 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 19:00:23.993584 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:23.993547 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:00:39.793824 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:39.793712 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:00:52.792661 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:00:52.792627 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:01:32.296200 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:32.296159 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:01:35.944856 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:35.944824 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4"] Jul 10 19:01:35.945253 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:35.945010 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" podUID="7e1b9943-685d-49f6-8096-85d95d293155" containerName="istio-proxy" containerID="cri-o://ace421984988ba73f80c4a1b87cb5963b812000df6d00d45662d857567651d7a" gracePeriod=30 Jul 10 19:01:41.199476 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.199453 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 19:01:41.255799 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.255713 2576 generic.go:356] "Generic (PLEG): container finished" podID="7e1b9943-685d-49f6-8096-85d95d293155" containerID="ace421984988ba73f80c4a1b87cb5963b812000df6d00d45662d857567651d7a" exitCode=0 Jul 10 19:01:41.255799 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.255761 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" Jul 10 19:01:41.255968 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.255763 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" event={"ID":"7e1b9943-685d-49f6-8096-85d95d293155","Type":"ContainerDied","Data":"ace421984988ba73f80c4a1b87cb5963b812000df6d00d45662d857567651d7a"} Jul 10 19:01:41.255968 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.255887 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4" event={"ID":"7e1b9943-685d-49f6-8096-85d95d293155","Type":"ContainerDied","Data":"084d4bdeb911fc86046c9d8d42ede878bb4dcd90309efd79bee69581a7fceac5"} Jul 10 19:01:41.255968 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.255916 2576 scope.go:117] "RemoveContainer" containerID="ace421984988ba73f80c4a1b87cb5963b812000df6d00d45662d857567651d7a" Jul 10 19:01:41.264567 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.264550 2576 scope.go:117] "RemoveContainer" containerID="7c5041d07cc625bd0e83f43ee58a587414a5e59ad9948f316b59f1a9980a216c" Jul 10 19:01:41.272487 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.272469 2576 scope.go:117] "RemoveContainer" containerID="ace421984988ba73f80c4a1b87cb5963b812000df6d00d45662d857567651d7a" Jul 10 19:01:41.272729 ip-10-0-140-180 kubenswrapper[2576]: E0710 19:01:41.272713 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace421984988ba73f80c4a1b87cb5963b812000df6d00d45662d857567651d7a\": container with ID starting with ace421984988ba73f80c4a1b87cb5963b812000df6d00d45662d857567651d7a not found: ID does not exist" containerID="ace421984988ba73f80c4a1b87cb5963b812000df6d00d45662d857567651d7a" Jul 10 19:01:41.272774 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.272737 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace421984988ba73f80c4a1b87cb5963b812000df6d00d45662d857567651d7a"} err="failed to get container status \"ace421984988ba73f80c4a1b87cb5963b812000df6d00d45662d857567651d7a\": rpc error: code = NotFound desc = could not find container \"ace421984988ba73f80c4a1b87cb5963b812000df6d00d45662d857567651d7a\": container with ID starting with ace421984988ba73f80c4a1b87cb5963b812000df6d00d45662d857567651d7a not found: ID does not exist" Jul 10 19:01:41.272774 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.272754 2576 scope.go:117] "RemoveContainer" containerID="7c5041d07cc625bd0e83f43ee58a587414a5e59ad9948f316b59f1a9980a216c" Jul 10 19:01:41.273016 ip-10-0-140-180 kubenswrapper[2576]: E0710 19:01:41.272999 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5041d07cc625bd0e83f43ee58a587414a5e59ad9948f316b59f1a9980a216c\": container with ID starting with 7c5041d07cc625bd0e83f43ee58a587414a5e59ad9948f316b59f1a9980a216c not found: ID does not exist" containerID="7c5041d07cc625bd0e83f43ee58a587414a5e59ad9948f316b59f1a9980a216c" Jul 10 19:01:41.273069 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.273022 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5041d07cc625bd0e83f43ee58a587414a5e59ad9948f316b59f1a9980a216c"} err="failed to get container status \"7c5041d07cc625bd0e83f43ee58a587414a5e59ad9948f316b59f1a9980a216c\": rpc error: code = NotFound desc = could not find container \"7c5041d07cc625bd0e83f43ee58a587414a5e59ad9948f316b59f1a9980a216c\": container with ID starting with 7c5041d07cc625bd0e83f43ee58a587414a5e59ad9948f316b59f1a9980a216c not found: ID does not exist" Jul 10 19:01:41.385362 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.385333 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1b9943-685d-49f6-8096-85d95d293155-istiod-ca-cert\") pod \"7e1b9943-685d-49f6-8096-85d95d293155\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " Jul 10 19:01:41.385536 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.385406 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-workload-socket\") pod \"7e1b9943-685d-49f6-8096-85d95d293155\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " Jul 10 19:01:41.385536 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.385469 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1b9943-685d-49f6-8096-85d95d293155-istio-token\") pod \"7e1b9943-685d-49f6-8096-85d95d293155\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " Jul 10 19:01:41.385536 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.385490 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fk66\" (UniqueName: \"kubernetes.io/projected/7e1b9943-685d-49f6-8096-85d95d293155-kube-api-access-4fk66\") pod \"7e1b9943-685d-49f6-8096-85d95d293155\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " Jul 10 19:01:41.385536 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.385529 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-credential-socket\") pod \"7e1b9943-685d-49f6-8096-85d95d293155\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " Jul 10 19:01:41.385754 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.385546 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-istio-envoy\") pod \"7e1b9943-685d-49f6-8096-85d95d293155\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " Jul 10 19:01:41.385754 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.385571 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1b9943-685d-49f6-8096-85d95d293155-istio-podinfo\") pod \"7e1b9943-685d-49f6-8096-85d95d293155\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " Jul 10 19:01:41.385754 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.385605 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-istio-data\") pod \"7e1b9943-685d-49f6-8096-85d95d293155\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " Jul 10 19:01:41.385754 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.385633 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-workload-certs\") pod \"7e1b9943-685d-49f6-8096-85d95d293155\" (UID: \"7e1b9943-685d-49f6-8096-85d95d293155\") " Jul 10 19:01:41.385754 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.385698 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e1b9943-685d-49f6-8096-85d95d293155" (UID: "7e1b9943-685d-49f6-8096-85d95d293155"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:01:41.385754 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.385710 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1b9943-685d-49f6-8096-85d95d293155-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e1b9943-685d-49f6-8096-85d95d293155" (UID: "7e1b9943-685d-49f6-8096-85d95d293155"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:01:41.386075 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.385851 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e1b9943-685d-49f6-8096-85d95d293155" (UID: "7e1b9943-685d-49f6-8096-85d95d293155"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:01:41.386075 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.385946 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-credential-socket\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:01:41.386075 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.385959 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1b9943-685d-49f6-8096-85d95d293155-istiod-ca-cert\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:01:41.386075 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.385970 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-workload-socket\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:01:41.386335 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.386182 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e1b9943-685d-49f6-8096-85d95d293155" (UID: "7e1b9943-685d-49f6-8096-85d95d293155"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:01:41.387191 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.387150 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e1b9943-685d-49f6-8096-85d95d293155" (UID: "7e1b9943-685d-49f6-8096-85d95d293155"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:01:41.388098 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.388047 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e1b9943-685d-49f6-8096-85d95d293155-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e1b9943-685d-49f6-8096-85d95d293155" (UID: "7e1b9943-685d-49f6-8096-85d95d293155"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:01:41.388201 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.388142 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e1b9943-685d-49f6-8096-85d95d293155" (UID: "7e1b9943-685d-49f6-8096-85d95d293155"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:01:41.388201 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.388183 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1b9943-685d-49f6-8096-85d95d293155-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e1b9943-685d-49f6-8096-85d95d293155" (UID: "7e1b9943-685d-49f6-8096-85d95d293155"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:01:41.388201 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.388185 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1b9943-685d-49f6-8096-85d95d293155-kube-api-access-4fk66" (OuterVolumeSpecName: "kube-api-access-4fk66") pod "7e1b9943-685d-49f6-8096-85d95d293155" (UID: "7e1b9943-685d-49f6-8096-85d95d293155"). InnerVolumeSpecName "kube-api-access-4fk66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:01:41.487427 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.487396 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-workload-certs\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:01:41.487427 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.487420 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1b9943-685d-49f6-8096-85d95d293155-istio-token\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:01:41.487427 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.487429 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fk66\" (UniqueName: \"kubernetes.io/projected/7e1b9943-685d-49f6-8096-85d95d293155-kube-api-access-4fk66\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:01:41.487654 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.487440 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-istio-envoy\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:01:41.487654 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.487449 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1b9943-685d-49f6-8096-85d95d293155-istio-podinfo\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:01:41.487654 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.487457 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1b9943-685d-49f6-8096-85d95d293155-istio-data\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:01:41.578346 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.578316 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4"] Jul 10 19:01:41.586106 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.586076 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b39babdc-openshift-default-5f75dcdbcf-87dc4"] Jul 10 19:01:41.843903 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:01:41.843859 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e1b9943-685d-49f6-8096-85d95d293155" path="/var/lib/kubelet/pods/7e1b9943-685d-49f6-8096-85d95d293155/volumes" Jul 10 19:02:31.320067 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.320022 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52"] Jul 10 19:02:31.320672 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.320547 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e1b9943-685d-49f6-8096-85d95d293155" containerName="istio-proxy" Jul 10 19:02:31.320672 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.320566 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e1b9943-685d-49f6-8096-85d95d293155" containerName="istio-proxy" Jul 10 19:02:31.320672 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.320581 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e1b9943-685d-49f6-8096-85d95d293155" containerName="istio-proxy" Jul 10 19:02:31.320672 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.320592 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e1b9943-685d-49f6-8096-85d95d293155" containerName="istio-proxy" Jul 10 19:02:31.320853 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.320704 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e1b9943-685d-49f6-8096-85d95d293155" containerName="istio-proxy" Jul 10 19:02:31.320853 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.320715 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e1b9943-685d-49f6-8096-85d95d293155" containerName="istio-proxy" Jul 10 19:02:31.324096 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.324077 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.326527 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.326509 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6b8252-openshift-default-dockercfg-wckhw\"" Jul 10 19:02:31.334814 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.334794 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52"] Jul 10 19:02:31.407429 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.407394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-envoy\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.407429 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.407433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-podinfo\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.407756 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.407463 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-credential-socket\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.407756 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.407498 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-token\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.407756 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.407600 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-workload-certs\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.407756 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.407666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istiod-ca-cert\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.407756 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.407703 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-data\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.407756 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.407729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qbc\" (UniqueName: \"kubernetes.io/projected/7b1da37d-3a0e-4d21-b53c-344f10c806f2-kube-api-access-49qbc\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.408075 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.407774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-workload-socket\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.508504 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.508474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-envoy\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.508504 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.508512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-podinfo\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.508981 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.508546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-credential-socket\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.508981 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.508590 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-token\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.508981 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.508629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-workload-certs\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.508981 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.508655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istiod-ca-cert\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.508981 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.508679 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-data\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.508981 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.508709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qbc\" (UniqueName: \"kubernetes.io/projected/7b1da37d-3a0e-4d21-b53c-344f10c806f2-kube-api-access-49qbc\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.508981 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.508748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-workload-socket\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.509390 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.509099 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-workload-socket\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.509390 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.509295 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-workload-certs\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.509390 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.509368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-data\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.509643 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.509623 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-credential-socket\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.510065 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.510041 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istiod-ca-cert\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.511118 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.511095 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-envoy\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.512607 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.512580 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-podinfo\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.517223 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.517204 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-token\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.517634 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.517614 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qbc\" (UniqueName: \"kubernetes.io/projected/7b1da37d-3a0e-4d21-b53c-344f10c806f2-kube-api-access-49qbc\") pod \"e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.636234 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.636143 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:31.774118 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.774088 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52"] Jul 10 19:02:31.775294 ip-10-0-140-180 kubenswrapper[2576]: W0710 19:02:31.775269 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1da37d_3a0e_4d21_b53c_344f10c806f2.slice/crio-bce1ec365ac60afe3c9fdc47c12797e268d20c283623031f689edeb6bd595c6a WatchSource:0}: Error finding container bce1ec365ac60afe3c9fdc47c12797e268d20c283623031f689edeb6bd595c6a: Status 404 returned error can't find the container with id bce1ec365ac60afe3c9fdc47c12797e268d20c283623031f689edeb6bd595c6a Jul 10 19:02:31.777592 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.777556 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 19:02:31.777657 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.777625 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 19:02:31.777695 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:31.777655 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 19:02:32.529285 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:32.529251 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" event={"ID":"7b1da37d-3a0e-4d21-b53c-344f10c806f2","Type":"ContainerStarted","Data":"aa5513de4fa48c6a9a736f6b58fd27719290f4560c4eb6afcd60aaba74147187"} Jul 10 19:02:32.529694 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:32.529312 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" event={"ID":"7b1da37d-3a0e-4d21-b53c-344f10c806f2","Type":"ContainerStarted","Data":"bce1ec365ac60afe3c9fdc47c12797e268d20c283623031f689edeb6bd595c6a"} Jul 10 19:02:32.550304 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:32.550260 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" podStartSLOduration=1.5502446810000001 podStartE2EDuration="1.550244681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:02:32.547509015 +0000 UTC m=+1999.225221479" watchObservedRunningTime="2026-07-10 19:02:32.550244681 +0000 UTC m=+1999.227957123" Jul 10 19:02:32.637183 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:32.637149 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:32.642327 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:32.642303 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:33.534764 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:33.534730 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:33.535828 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:33.535803 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:42.693082 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:42.693047 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:02:49.535740 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:49.535692 2576 patch_prober.go:28] interesting pod/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 19:02:49.536140 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:49.535756 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" podUID="7b1da37d-3a0e-4d21-b53c-344f10c806f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 19:02:50.624328 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:50.624296 2576 generic.go:356] "Generic (PLEG): container finished" podID="7b1da37d-3a0e-4d21-b53c-344f10c806f2" containerID="aa5513de4fa48c6a9a736f6b58fd27719290f4560c4eb6afcd60aaba74147187" exitCode=0 Jul 10 19:02:50.624734 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:50.624335 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" event={"ID":"7b1da37d-3a0e-4d21-b53c-344f10c806f2","Type":"ContainerDied","Data":"aa5513de4fa48c6a9a736f6b58fd27719290f4560c4eb6afcd60aaba74147187"} Jul 10 19:02:50.624734 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:50.624709 2576 scope.go:117] "RemoveContainer" containerID="aa5513de4fa48c6a9a736f6b58fd27719290f4560c4eb6afcd60aaba74147187" Jul 10 19:02:50.625772 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:50.625742 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 19:02:50.625873 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:50.625808 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 19:02:50.625873 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:50.625838 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 10 19:02:51.634011 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:51.633972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" event={"ID":"7b1da37d-3a0e-4d21-b53c-344f10c806f2","Type":"ContainerStarted","Data":"c61b980de3c3e3c1df8e8bd717116a32244dad2b64d931f4096911f5a3d6eadc"} Jul 10 19:02:51.636813 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:51.636784 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:51.637159 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:51.637133 2576 patch_prober.go:28] interesting pod/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 10 19:02:51.637295 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:51.637190 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" podUID="7b1da37d-3a0e-4d21-b53c-344f10c806f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 10 19:02:52.636955 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:52.636919 2576 patch_prober.go:28] interesting pod/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 10 19:02:52.637648 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:52.637617 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" podUID="7b1da37d-3a0e-4d21-b53c-344f10c806f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 10 19:02:53.636568 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:53.636528 2576 patch_prober.go:28] interesting pod/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 10 19:02:53.636792 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:53.636601 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" podUID="7b1da37d-3a0e-4d21-b53c-344f10c806f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 10 19:02:54.637559 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:54.637523 2576 patch_prober.go:28] interesting pod/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 10 19:02:54.638139 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:54.638107 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" podUID="7b1da37d-3a0e-4d21-b53c-344f10c806f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 10 19:02:55.640039 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:55.640006 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:55.640433 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:55.640379 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:55.641146 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:55.641125 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:02:58.289269 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:02:58.289234 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:03:16.087332 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.087291 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg"] Jul 10 19:03:16.090241 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.090221 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.092785 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.092764 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-12e952\"/\"openshift-service-ca.crt\"" Jul 10 19:03:16.093737 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.093722 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-12e952\"/\"test-model-12e952-kserve-self-signed-certs\"" Jul 10 19:03:16.093843 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.093800 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-12e952\"/\"default-dockercfg-hqn48\"" Jul 10 19:03:16.093843 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.093818 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-12e952\"/\"kube-root-ca.crt\"" Jul 10 19:03:16.100475 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.100448 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg"] Jul 10 19:03:16.206519 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.206489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-model-cache\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.206727 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.206533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-home\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.206727 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.206558 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v69vw\" (UniqueName: \"kubernetes.io/projected/44befc58-3eef-4f17-94ed-388c7479b27e-kube-api-access-v69vw\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.206727 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.206676 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44befc58-3eef-4f17-94ed-388c7479b27e-tls-certs\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.206897 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.206757 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-dshm\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.206897 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.206783 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-tmp-dir\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.307801 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.307760 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44befc58-3eef-4f17-94ed-388c7479b27e-tls-certs\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.307986 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.307821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-dshm\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.307986 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.307842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-tmp-dir\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.307986 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.307880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-model-cache\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.308126 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.308005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-home\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.308126 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.308043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v69vw\" (UniqueName: \"kubernetes.io/projected/44befc58-3eef-4f17-94ed-388c7479b27e-kube-api-access-v69vw\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.308392 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.308344 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-tmp-dir\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.308495 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.308400 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-model-cache\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.308495 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.308441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-home\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.310166 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.310148 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-dshm\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.310590 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.310571 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44befc58-3eef-4f17-94ed-388c7479b27e-tls-certs\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.315905 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.315882 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v69vw\" (UniqueName: \"kubernetes.io/projected/44befc58-3eef-4f17-94ed-388c7479b27e-kube-api-access-v69vw\") pod \"test-model-12e952-kserve-56f569597c-sfgtg\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.400151 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.400064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:16.536614 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.536583 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg"] Jul 10 19:03:16.540054 ip-10-0-140-180 kubenswrapper[2576]: W0710 19:03:16.540024 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44befc58_3eef_4f17_94ed_388c7479b27e.slice/crio-b9a174a352529f2c94e8c2d1789094b9daecf0ef0e1173529d2c622a1b01c655 WatchSource:0}: Error finding container b9a174a352529f2c94e8c2d1789094b9daecf0ef0e1173529d2c622a1b01c655: Status 404 returned error can't find the container with id b9a174a352529f2c94e8c2d1789094b9daecf0ef0e1173529d2c622a1b01c655 Jul 10 19:03:16.777409 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.777298 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" event={"ID":"44befc58-3eef-4f17-94ed-388c7479b27e","Type":"ContainerStarted","Data":"2cbda07723b388453a4c476fba7ddb8a4e92dc699d4a5f842f565e68348de8a6"} Jul 10 19:03:16.777409 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.777339 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" event={"ID":"44befc58-3eef-4f17-94ed-388c7479b27e","Type":"ContainerStarted","Data":"b9a174a352529f2c94e8c2d1789094b9daecf0ef0e1173529d2c622a1b01c655"} Jul 10 19:03:16.795132 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:16.795078 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" podStartSLOduration=0.795062921 podStartE2EDuration="795.062921ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:03:16.792538027 +0000 UTC m=+2043.470250490" watchObservedRunningTime="2026-07-10 19:03:16.795062921 +0000 UTC m=+2043.472775382" Jul 10 19:03:20.782806 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:20.782773 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:03:26.400443 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:26.400410 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:26.400905 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:26.400465 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:26.413050 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:26.413017 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:26.842413 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:26.842382 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:33.728571 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:33.728540 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg"] Jul 10 19:03:38.987271 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:38.987187 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:03:48.749395 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:48.749303 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" podUID="44befc58-3eef-4f17-94ed-388c7479b27e" containerName="main" containerID="cri-o://2cbda07723b388453a4c476fba7ddb8a4e92dc699d4a5f842f565e68348de8a6" gracePeriod=45 Jul 10 19:03:48.947144 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:48.947114 2576 generic.go:356] "Generic (PLEG): container finished" podID="44befc58-3eef-4f17-94ed-388c7479b27e" containerID="2cbda07723b388453a4c476fba7ddb8a4e92dc699d4a5f842f565e68348de8a6" exitCode=0 Jul 10 19:03:48.947294 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:48.947155 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" event={"ID":"44befc58-3eef-4f17-94ed-388c7479b27e","Type":"ContainerDied","Data":"2cbda07723b388453a4c476fba7ddb8a4e92dc699d4a5f842f565e68348de8a6"} Jul 10 19:03:49.000700 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.000639 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:49.093305 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.093275 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v69vw\" (UniqueName: \"kubernetes.io/projected/44befc58-3eef-4f17-94ed-388c7479b27e-kube-api-access-v69vw\") pod \"44befc58-3eef-4f17-94ed-388c7479b27e\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " Jul 10 19:03:49.093507 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.093331 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-tmp-dir\") pod \"44befc58-3eef-4f17-94ed-388c7479b27e\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " Jul 10 19:03:49.093507 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.093374 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-dshm\") pod \"44befc58-3eef-4f17-94ed-388c7479b27e\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " Jul 10 19:03:49.093507 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.093392 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-model-cache\") pod \"44befc58-3eef-4f17-94ed-388c7479b27e\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " Jul 10 19:03:49.093507 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.093414 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-home\") pod \"44befc58-3eef-4f17-94ed-388c7479b27e\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " Jul 10 19:03:49.093507 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.093455 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44befc58-3eef-4f17-94ed-388c7479b27e-tls-certs\") pod \"44befc58-3eef-4f17-94ed-388c7479b27e\" (UID: \"44befc58-3eef-4f17-94ed-388c7479b27e\") " Jul 10 19:03:49.093755 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.093596 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44befc58-3eef-4f17-94ed-388c7479b27e" (UID: "44befc58-3eef-4f17-94ed-388c7479b27e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:03:49.093755 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.093676 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-model-cache" (OuterVolumeSpecName: "model-cache") pod "44befc58-3eef-4f17-94ed-388c7479b27e" (UID: "44befc58-3eef-4f17-94ed-388c7479b27e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:03:49.093755 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.093694 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-home" (OuterVolumeSpecName: "home") pod "44befc58-3eef-4f17-94ed-388c7479b27e" (UID: "44befc58-3eef-4f17-94ed-388c7479b27e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:03:49.095564 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.095539 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-dshm" (OuterVolumeSpecName: "dshm") pod "44befc58-3eef-4f17-94ed-388c7479b27e" (UID: "44befc58-3eef-4f17-94ed-388c7479b27e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:03:49.095683 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.095663 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44befc58-3eef-4f17-94ed-388c7479b27e-kube-api-access-v69vw" (OuterVolumeSpecName: "kube-api-access-v69vw") pod "44befc58-3eef-4f17-94ed-388c7479b27e" (UID: "44befc58-3eef-4f17-94ed-388c7479b27e"). InnerVolumeSpecName "kube-api-access-v69vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:03:49.095752 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.095692 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44befc58-3eef-4f17-94ed-388c7479b27e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44befc58-3eef-4f17-94ed-388c7479b27e" (UID: "44befc58-3eef-4f17-94ed-388c7479b27e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:03:49.194386 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.194336 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v69vw\" (UniqueName: \"kubernetes.io/projected/44befc58-3eef-4f17-94ed-388c7479b27e-kube-api-access-v69vw\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:03:49.194386 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.194384 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-tmp-dir\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:03:49.194386 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.194396 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-dshm\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:03:49.194644 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.194404 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-model-cache\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:03:49.194644 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.194413 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44befc58-3eef-4f17-94ed-388c7479b27e-home\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:03:49.194644 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.194422 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44befc58-3eef-4f17-94ed-388c7479b27e-tls-certs\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:03:49.953857 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.953825 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" Jul 10 19:03:49.954275 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.953858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg" event={"ID":"44befc58-3eef-4f17-94ed-388c7479b27e","Type":"ContainerDied","Data":"b9a174a352529f2c94e8c2d1789094b9daecf0ef0e1173529d2c622a1b01c655"} Jul 10 19:03:49.954275 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.953907 2576 scope.go:117] "RemoveContainer" containerID="2cbda07723b388453a4c476fba7ddb8a4e92dc699d4a5f842f565e68348de8a6" Jul 10 19:03:49.970558 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.970535 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg"] Jul 10 19:03:49.976532 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:49.976505 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-12e952/test-model-12e952-kserve-56f569597c-sfgtg"] Jul 10 19:03:51.843141 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:03:51.843109 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44befc58-3eef-4f17-94ed-388c7479b27e" path="/var/lib/kubelet/pods/44befc58-3eef-4f17-94ed-388c7479b27e/volumes" Jul 10 19:04:13.943969 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:13.943940 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 19:04:13.949152 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:13.949129 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 19:04:16.460991 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:16.460959 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52"] Jul 10 19:04:16.461475 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:16.461213 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" podUID="7b1da37d-3a0e-4d21-b53c-344f10c806f2" containerName="istio-proxy" containerID="cri-o://c61b980de3c3e3c1df8e8bd717116a32244dad2b64d931f4096911f5a3d6eadc" gracePeriod=30 Jul 10 19:04:21.213893 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.213856 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:04:21.727663 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.727607 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:04:21.795458 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.795424 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-podinfo\") pod \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " Jul 10 19:04:21.795645 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.795466 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-workload-certs\") pod \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " Jul 10 19:04:21.795645 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.795512 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-workload-socket\") pod \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " Jul 10 19:04:21.795645 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.795541 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-credential-socket\") pod \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " Jul 10 19:04:21.795645 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.795623 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istiod-ca-cert\") pod \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " Jul 10 19:04:21.795885 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.795679 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-token\") pod \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " Jul 10 19:04:21.795885 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.795712 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-envoy\") pod \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " Jul 10 19:04:21.795885 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.795741 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-data\") pod \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " Jul 10 19:04:21.795885 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.795766 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49qbc\" (UniqueName: \"kubernetes.io/projected/7b1da37d-3a0e-4d21-b53c-344f10c806f2-kube-api-access-49qbc\") pod \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\" (UID: \"7b1da37d-3a0e-4d21-b53c-344f10c806f2\") " Jul 10 19:04:21.796083 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.795907 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b1da37d-3a0e-4d21-b53c-344f10c806f2" (UID: "7b1da37d-3a0e-4d21-b53c-344f10c806f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:21.796083 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.795919 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b1da37d-3a0e-4d21-b53c-344f10c806f2" (UID: "7b1da37d-3a0e-4d21-b53c-344f10c806f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:21.796083 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.795954 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b1da37d-3a0e-4d21-b53c-344f10c806f2" (UID: "7b1da37d-3a0e-4d21-b53c-344f10c806f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:21.796244 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.796103 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-credential-socket\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:04:21.796244 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.796121 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-workload-certs\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:04:21.796244 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.796134 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-workload-socket\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:04:21.796244 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.796088 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b1da37d-3a0e-4d21-b53c-344f10c806f2" (UID: "7b1da37d-3a0e-4d21-b53c-344f10c806f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:04:21.796991 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.796961 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b1da37d-3a0e-4d21-b53c-344f10c806f2" (UID: "7b1da37d-3a0e-4d21-b53c-344f10c806f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:21.797834 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.797806 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b1da37d-3a0e-4d21-b53c-344f10c806f2" (UID: "7b1da37d-3a0e-4d21-b53c-344f10c806f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:21.797834 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.797804 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b1da37d-3a0e-4d21-b53c-344f10c806f2" (UID: "7b1da37d-3a0e-4d21-b53c-344f10c806f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:04:21.798020 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.797834 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b1da37d-3a0e-4d21-b53c-344f10c806f2" (UID: "7b1da37d-3a0e-4d21-b53c-344f10c806f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:04:21.798020 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.797906 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1da37d-3a0e-4d21-b53c-344f10c806f2-kube-api-access-49qbc" (OuterVolumeSpecName: "kube-api-access-49qbc") pod "7b1da37d-3a0e-4d21-b53c-344f10c806f2" (UID: "7b1da37d-3a0e-4d21-b53c-344f10c806f2"). InnerVolumeSpecName "kube-api-access-49qbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:04:21.896487 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.896457 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istiod-ca-cert\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:04:21.896487 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.896480 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-token\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:04:21.896487 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.896489 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-envoy\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:04:21.896688 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.896497 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-data\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:04:21.896688 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.896505 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49qbc\" (UniqueName: \"kubernetes.io/projected/7b1da37d-3a0e-4d21-b53c-344f10c806f2-kube-api-access-49qbc\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:04:21.896688 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:21.896514 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1da37d-3a0e-4d21-b53c-344f10c806f2-istio-podinfo\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:04:22.127014 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:22.126979 2576 generic.go:356] "Generic (PLEG): container finished" podID="7b1da37d-3a0e-4d21-b53c-344f10c806f2" containerID="c61b980de3c3e3c1df8e8bd717116a32244dad2b64d931f4096911f5a3d6eadc" exitCode=0 Jul 10 19:04:22.127208 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:22.127028 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" Jul 10 19:04:22.127208 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:22.127060 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" event={"ID":"7b1da37d-3a0e-4d21-b53c-344f10c806f2","Type":"ContainerDied","Data":"c61b980de3c3e3c1df8e8bd717116a32244dad2b64d931f4096911f5a3d6eadc"} Jul 10 19:04:22.127208 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:22.127120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52" event={"ID":"7b1da37d-3a0e-4d21-b53c-344f10c806f2","Type":"ContainerDied","Data":"bce1ec365ac60afe3c9fdc47c12797e268d20c283623031f689edeb6bd595c6a"} Jul 10 19:04:22.127208 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:22.127150 2576 scope.go:117] "RemoveContainer" containerID="c61b980de3c3e3c1df8e8bd717116a32244dad2b64d931f4096911f5a3d6eadc" Jul 10 19:04:22.136480 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:22.136460 2576 scope.go:117] "RemoveContainer" containerID="aa5513de4fa48c6a9a736f6b58fd27719290f4560c4eb6afcd60aaba74147187" Jul 10 19:04:22.144871 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:22.144850 2576 scope.go:117] "RemoveContainer" containerID="c61b980de3c3e3c1df8e8bd717116a32244dad2b64d931f4096911f5a3d6eadc" Jul 10 19:04:22.145120 ip-10-0-140-180 kubenswrapper[2576]: E0710 19:04:22.145100 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61b980de3c3e3c1df8e8bd717116a32244dad2b64d931f4096911f5a3d6eadc\": container with ID starting with c61b980de3c3e3c1df8e8bd717116a32244dad2b64d931f4096911f5a3d6eadc not found: ID does not exist" containerID="c61b980de3c3e3c1df8e8bd717116a32244dad2b64d931f4096911f5a3d6eadc" Jul 10 19:04:22.145180 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:22.145127 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61b980de3c3e3c1df8e8bd717116a32244dad2b64d931f4096911f5a3d6eadc"} err="failed to get container status \"c61b980de3c3e3c1df8e8bd717116a32244dad2b64d931f4096911f5a3d6eadc\": rpc error: code = NotFound desc = could not find container \"c61b980de3c3e3c1df8e8bd717116a32244dad2b64d931f4096911f5a3d6eadc\": container with ID starting with c61b980de3c3e3c1df8e8bd717116a32244dad2b64d931f4096911f5a3d6eadc not found: ID does not exist" Jul 10 19:04:22.145180 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:22.145146 2576 scope.go:117] "RemoveContainer" containerID="aa5513de4fa48c6a9a736f6b58fd27719290f4560c4eb6afcd60aaba74147187" Jul 10 19:04:22.145400 ip-10-0-140-180 kubenswrapper[2576]: E0710 19:04:22.145341 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5513de4fa48c6a9a736f6b58fd27719290f4560c4eb6afcd60aaba74147187\": container with ID starting with aa5513de4fa48c6a9a736f6b58fd27719290f4560c4eb6afcd60aaba74147187 not found: ID does not exist" containerID="aa5513de4fa48c6a9a736f6b58fd27719290f4560c4eb6afcd60aaba74147187" Jul 10 19:04:22.145469 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:22.145412 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5513de4fa48c6a9a736f6b58fd27719290f4560c4eb6afcd60aaba74147187"} err="failed to get container status \"aa5513de4fa48c6a9a736f6b58fd27719290f4560c4eb6afcd60aaba74147187\": rpc error: code = NotFound desc = could not find container \"aa5513de4fa48c6a9a736f6b58fd27719290f4560c4eb6afcd60aaba74147187\": container with ID starting with aa5513de4fa48c6a9a736f6b58fd27719290f4560c4eb6afcd60aaba74147187 not found: ID does not exist" Jul 10 19:04:22.145822 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:22.145804 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52"] Jul 10 19:04:22.149773 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:22.149750 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6b8252-openshift-default-85bc7896c5-mnx52"] Jul 10 19:04:23.843141 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:23.843111 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b1da37d-3a0e-4d21-b53c-344f10c806f2" path="/var/lib/kubelet/pods/7b1da37d-3a0e-4d21-b53c-344f10c806f2/volumes" Jul 10 19:04:36.005558 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:36.005521 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:04:52.181337 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:04:52.181301 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:05:04.689261 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:04.689228 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:05:13.900384 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:13.900337 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:05:25.288927 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:25.288889 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:05:35.594175 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.594143 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99"] Jul 10 19:05:35.594680 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.594654 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1da37d-3a0e-4d21-b53c-344f10c806f2" containerName="istio-proxy" Jul 10 19:05:35.594680 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.594677 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1da37d-3a0e-4d21-b53c-344f10c806f2" containerName="istio-proxy" Jul 10 19:05:35.594880 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.594691 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1da37d-3a0e-4d21-b53c-344f10c806f2" containerName="istio-proxy" Jul 10 19:05:35.594880 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.594700 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1da37d-3a0e-4d21-b53c-344f10c806f2" containerName="istio-proxy" Jul 10 19:05:35.594880 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.594729 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44befc58-3eef-4f17-94ed-388c7479b27e" containerName="main" Jul 10 19:05:35.594880 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.594738 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44befc58-3eef-4f17-94ed-388c7479b27e" containerName="main" Jul 10 19:05:35.594880 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.594876 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1da37d-3a0e-4d21-b53c-344f10c806f2" containerName="istio-proxy" Jul 10 19:05:35.595135 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.594893 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44befc58-3eef-4f17-94ed-388c7479b27e" containerName="main" Jul 10 19:05:35.597902 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.597882 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.600438 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.600419 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-33ca20\"/\"kube-root-ca.crt\"" Jul 10 19:05:35.601196 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.601178 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-33ca20\"/\"default-dockercfg-wvv54\"" Jul 10 19:05:35.601270 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.601247 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-33ca20\"/\"auth-test-model-33ca20-kserve-self-signed-certs\"" Jul 10 19:05:35.601320 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.601276 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-33ca20\"/\"openshift-service-ca.crt\"" Jul 10 19:05:35.607847 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.607826 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99"] Jul 10 19:05:35.711380 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.711329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-tmp-dir\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.711577 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.711425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f41a8f1-9be4-439a-ab32-5ed564583d97-tls-certs\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.711577 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.711479 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7m4q\" (UniqueName: \"kubernetes.io/projected/8f41a8f1-9be4-439a-ab32-5ed564583d97-kube-api-access-n7m4q\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.711577 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.711551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-model-cache\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.711715 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.711591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-home\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.711715 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.711618 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-dshm\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.812602 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.812567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-model-cache\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.812764 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.812608 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-home\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.812764 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.812653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-dshm\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.812764 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.812725 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-tmp-dir\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.812764 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.812752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f41a8f1-9be4-439a-ab32-5ed564583d97-tls-certs\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.813045 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.812775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7m4q\" (UniqueName: \"kubernetes.io/projected/8f41a8f1-9be4-439a-ab32-5ed564583d97-kube-api-access-n7m4q\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.813045 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.813000 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-model-cache\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.813159 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.813061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-tmp-dir\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.813207 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.813157 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-home\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.814925 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.814901 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-dshm\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.815173 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.815156 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f41a8f1-9be4-439a-ab32-5ed564583d97-tls-certs\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.822746 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.822726 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7m4q\" (UniqueName: \"kubernetes.io/projected/8f41a8f1-9be4-439a-ab32-5ed564583d97-kube-api-access-n7m4q\") pod \"auth-test-model-33ca20-kserve-7d594f46b7-l4m99\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:35.908046 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:35.907943 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:36.051088 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:36.050972 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99"] Jul 10 19:05:36.053796 ip-10-0-140-180 kubenswrapper[2576]: W0710 19:05:36.053770 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f41a8f1_9be4_439a_ab32_5ed564583d97.slice/crio-5667ab78da4fae604a77146e32e9303e10f06274047d25a905805a5587035dd8 WatchSource:0}: Error finding container 5667ab78da4fae604a77146e32e9303e10f06274047d25a905805a5587035dd8: Status 404 returned error can't find the container with id 5667ab78da4fae604a77146e32e9303e10f06274047d25a905805a5587035dd8 Jul 10 19:05:36.507181 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:36.507144 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" event={"ID":"8f41a8f1-9be4-439a-ab32-5ed564583d97","Type":"ContainerStarted","Data":"f67086077db7ed616922aad302b1d9e95b779921f92fcddd968a41300128be35"} Jul 10 19:05:36.507372 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:36.507198 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" event={"ID":"8f41a8f1-9be4-439a-ab32-5ed564583d97","Type":"ContainerStarted","Data":"5667ab78da4fae604a77146e32e9303e10f06274047d25a905805a5587035dd8"} Jul 10 19:05:36.523981 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:36.523935 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" podStartSLOduration=1.523918561 podStartE2EDuration="1.523918561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:36.52246122 +0000 UTC m=+2183.200173694" watchObservedRunningTime="2026-07-10 19:05:36.523918561 +0000 UTC m=+2183.201631020" Jul 10 19:05:39.015223 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:39.015191 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:05:45.908256 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:45.908218 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:45.908669 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:45.908510 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:45.920774 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:45.920746 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:46.568380 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:46.568340 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:05:54.040335 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.040297 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c"] Jul 10 19:05:54.040710 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.040694 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1da37d-3a0e-4d21-b53c-344f10c806f2" containerName="istio-proxy" Jul 10 19:05:54.043461 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.043443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.047857 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.047836 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7b46a3\"/\"auth-test-model-7b46a3-kserve-self-signed-certs\"" Jul 10 19:05:54.048339 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.048325 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7b46a3\"/\"default-dockercfg-ql8fs\"" Jul 10 19:05:54.048514 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.048495 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7b46a3\"/\"kube-root-ca.crt\"" Jul 10 19:05:54.048792 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.048777 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7b46a3\"/\"openshift-service-ca.crt\"" Jul 10 19:05:54.072182 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.072155 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c"] Jul 10 19:05:54.175406 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.175334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-tmp-dir\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.175574 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.175420 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e481ac0-f528-4f3d-92b5-13b3541b23bb-tls-certs\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.175574 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.175482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-model-cache\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.175574 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.175561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flbbt\" (UniqueName: \"kubernetes.io/projected/6e481ac0-f528-4f3d-92b5-13b3541b23bb-kube-api-access-flbbt\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.175688 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.175615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-home\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.175688 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.175673 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-dshm\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.276823 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.276788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-dshm\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.277004 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.276870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-tmp-dir\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.277004 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.276956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e481ac0-f528-4f3d-92b5-13b3541b23bb-tls-certs\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.277137 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.277004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-model-cache\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.277137 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.277052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flbbt\" (UniqueName: \"kubernetes.io/projected/6e481ac0-f528-4f3d-92b5-13b3541b23bb-kube-api-access-flbbt\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.277137 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.277090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-home\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.277291 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.277261 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-tmp-dir\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.277424 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.277405 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-model-cache\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.277470 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.277446 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-home\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.278982 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.278960 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-dshm\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.279309 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.279292 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e481ac0-f528-4f3d-92b5-13b3541b23bb-tls-certs\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.286608 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.286589 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flbbt\" (UniqueName: \"kubernetes.io/projected/6e481ac0-f528-4f3d-92b5-13b3541b23bb-kube-api-access-flbbt\") pod \"auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.353405 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.353367 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:05:54.493621 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.493590 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c"] Jul 10 19:05:54.494862 ip-10-0-140-180 kubenswrapper[2576]: W0710 19:05:54.494829 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e481ac0_f528_4f3d_92b5_13b3541b23bb.slice/crio-20e03dbf8c7dc9b261097547ec1b467021cdb758bb1ccf04dc604f0d46452a5b WatchSource:0}: Error finding container 20e03dbf8c7dc9b261097547ec1b467021cdb758bb1ccf04dc604f0d46452a5b: Status 404 returned error can't find the container with id 20e03dbf8c7dc9b261097547ec1b467021cdb758bb1ccf04dc604f0d46452a5b Jul 10 19:05:54.606158 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.606074 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" event={"ID":"6e481ac0-f528-4f3d-92b5-13b3541b23bb","Type":"ContainerStarted","Data":"51ec27fc7ca12356f7ff0506a37264d9cf2abe11d5d6dc548d22055f51365fa2"} Jul 10 19:05:54.606158 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.606115 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" event={"ID":"6e481ac0-f528-4f3d-92b5-13b3541b23bb","Type":"ContainerStarted","Data":"20e03dbf8c7dc9b261097547ec1b467021cdb758bb1ccf04dc604f0d46452a5b"} Jul 10 19:05:54.627233 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:54.627183 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" podStartSLOduration=0.627168403 podStartE2EDuration="627.168403ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:54.624832847 +0000 UTC m=+2201.302545309" watchObservedRunningTime="2026-07-10 19:05:54.627168403 +0000 UTC m=+2201.304880864" Jul 10 19:05:57.893902 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:05:57.893865 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:06:04.353821 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:06:04.353780 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:06:04.353821 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:06:04.353832 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:06:04.366559 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:06:04.366530 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:06:04.674323 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:06:04.674239 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:06:18.792977 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:06:18.792944 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:06:38.093377 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:06:38.093276 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:06:51.898289 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:06:51.898253 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:07:08.692996 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:07:08.692960 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:07:27.494745 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:07:27.494707 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:07:44.989284 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:07:44.989253 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:07:58.111017 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:07:58.110982 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99"] Jul 10 19:07:59.069206 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:07:59.069166 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c"] Jul 10 19:08:02.585518 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:02.585481 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:08:13.130958 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.130907 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" podUID="8f41a8f1-9be4-439a-ab32-5ed564583d97" containerName="main" containerID="cri-o://f67086077db7ed616922aad302b1d9e95b779921f92fcddd968a41300128be35" gracePeriod=45 Jul 10 19:08:13.349069 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.349033 2576 generic.go:356] "Generic (PLEG): container finished" podID="8f41a8f1-9be4-439a-ab32-5ed564583d97" containerID="f67086077db7ed616922aad302b1d9e95b779921f92fcddd968a41300128be35" exitCode=0 Jul 10 19:08:13.349232 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.349098 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" event={"ID":"8f41a8f1-9be4-439a-ab32-5ed564583d97","Type":"ContainerDied","Data":"f67086077db7ed616922aad302b1d9e95b779921f92fcddd968a41300128be35"} Jul 10 19:08:13.372838 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.372808 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:08:13.379829 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.379808 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-model-cache\") pod \"8f41a8f1-9be4-439a-ab32-5ed564583d97\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " Jul 10 19:08:13.379920 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.379884 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-dshm\") pod \"8f41a8f1-9be4-439a-ab32-5ed564583d97\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " Jul 10 19:08:13.379959 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.379932 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-home\") pod \"8f41a8f1-9be4-439a-ab32-5ed564583d97\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " Jul 10 19:08:13.379996 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.379962 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f41a8f1-9be4-439a-ab32-5ed564583d97-tls-certs\") pod \"8f41a8f1-9be4-439a-ab32-5ed564583d97\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " Jul 10 19:08:13.379996 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.379983 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-tmp-dir\") pod \"8f41a8f1-9be4-439a-ab32-5ed564583d97\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " Jul 10 19:08:13.380097 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.380012 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7m4q\" (UniqueName: \"kubernetes.io/projected/8f41a8f1-9be4-439a-ab32-5ed564583d97-kube-api-access-n7m4q\") pod \"8f41a8f1-9be4-439a-ab32-5ed564583d97\" (UID: \"8f41a8f1-9be4-439a-ab32-5ed564583d97\") " Jul 10 19:08:13.380097 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.380068 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f41a8f1-9be4-439a-ab32-5ed564583d97" (UID: "8f41a8f1-9be4-439a-ab32-5ed564583d97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:08:13.380417 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.380247 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-home" (OuterVolumeSpecName: "home") pod "8f41a8f1-9be4-439a-ab32-5ed564583d97" (UID: "8f41a8f1-9be4-439a-ab32-5ed564583d97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:08:13.380417 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.380261 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f41a8f1-9be4-439a-ab32-5ed564583d97" (UID: "8f41a8f1-9be4-439a-ab32-5ed564583d97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:08:13.380417 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.380365 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-tmp-dir\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:08:13.380417 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.380384 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-model-cache\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:08:13.380417 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.380396 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-home\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:08:13.382107 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.382044 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-dshm" (OuterVolumeSpecName: "dshm") pod "8f41a8f1-9be4-439a-ab32-5ed564583d97" (UID: "8f41a8f1-9be4-439a-ab32-5ed564583d97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:08:13.382107 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.382057 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f41a8f1-9be4-439a-ab32-5ed564583d97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f41a8f1-9be4-439a-ab32-5ed564583d97" (UID: "8f41a8f1-9be4-439a-ab32-5ed564583d97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:08:13.382226 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.382170 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f41a8f1-9be4-439a-ab32-5ed564583d97-kube-api-access-n7m4q" (OuterVolumeSpecName: "kube-api-access-n7m4q") pod "8f41a8f1-9be4-439a-ab32-5ed564583d97" (UID: "8f41a8f1-9be4-439a-ab32-5ed564583d97"). InnerVolumeSpecName "kube-api-access-n7m4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:08:13.481637 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.481606 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f41a8f1-9be4-439a-ab32-5ed564583d97-dshm\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:08:13.481637 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.481632 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f41a8f1-9be4-439a-ab32-5ed564583d97-tls-certs\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:08:13.481637 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:13.481643 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7m4q\" (UniqueName: \"kubernetes.io/projected/8f41a8f1-9be4-439a-ab32-5ed564583d97-kube-api-access-n7m4q\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:08:14.089581 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.089540 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" podUID="6e481ac0-f528-4f3d-92b5-13b3541b23bb" containerName="main" containerID="cri-o://51ec27fc7ca12356f7ff0506a37264d9cf2abe11d5d6dc548d22055f51365fa2" gracePeriod=45 Jul 10 19:08:14.339304 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.339277 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:08:14.355435 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.355403 2576 generic.go:356] "Generic (PLEG): container finished" podID="6e481ac0-f528-4f3d-92b5-13b3541b23bb" containerID="51ec27fc7ca12356f7ff0506a37264d9cf2abe11d5d6dc548d22055f51365fa2" exitCode=0 Jul 10 19:08:14.355577 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.355509 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" Jul 10 19:08:14.355657 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.355510 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" event={"ID":"6e481ac0-f528-4f3d-92b5-13b3541b23bb","Type":"ContainerDied","Data":"51ec27fc7ca12356f7ff0506a37264d9cf2abe11d5d6dc548d22055f51365fa2"} Jul 10 19:08:14.355721 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.355692 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c" event={"ID":"6e481ac0-f528-4f3d-92b5-13b3541b23bb","Type":"ContainerDied","Data":"20e03dbf8c7dc9b261097547ec1b467021cdb758bb1ccf04dc604f0d46452a5b"} Jul 10 19:08:14.355790 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.355724 2576 scope.go:117] "RemoveContainer" containerID="51ec27fc7ca12356f7ff0506a37264d9cf2abe11d5d6dc548d22055f51365fa2" Jul 10 19:08:14.357514 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.357491 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" event={"ID":"8f41a8f1-9be4-439a-ab32-5ed564583d97","Type":"ContainerDied","Data":"5667ab78da4fae604a77146e32e9303e10f06274047d25a905805a5587035dd8"} Jul 10 19:08:14.357633 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.357536 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99" Jul 10 19:08:14.367633 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.367604 2576 scope.go:117] "RemoveContainer" containerID="51ec27fc7ca12356f7ff0506a37264d9cf2abe11d5d6dc548d22055f51365fa2" Jul 10 19:08:14.367904 ip-10-0-140-180 kubenswrapper[2576]: E0710 19:08:14.367880 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ec27fc7ca12356f7ff0506a37264d9cf2abe11d5d6dc548d22055f51365fa2\": container with ID starting with 51ec27fc7ca12356f7ff0506a37264d9cf2abe11d5d6dc548d22055f51365fa2 not found: ID does not exist" containerID="51ec27fc7ca12356f7ff0506a37264d9cf2abe11d5d6dc548d22055f51365fa2" Jul 10 19:08:14.367995 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.367913 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ec27fc7ca12356f7ff0506a37264d9cf2abe11d5d6dc548d22055f51365fa2"} err="failed to get container status \"51ec27fc7ca12356f7ff0506a37264d9cf2abe11d5d6dc548d22055f51365fa2\": rpc error: code = NotFound desc = could not find container \"51ec27fc7ca12356f7ff0506a37264d9cf2abe11d5d6dc548d22055f51365fa2\": container with ID starting with 51ec27fc7ca12356f7ff0506a37264d9cf2abe11d5d6dc548d22055f51365fa2 not found: ID does not exist" Jul 10 19:08:14.367995 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.367936 2576 scope.go:117] "RemoveContainer" containerID="f67086077db7ed616922aad302b1d9e95b779921f92fcddd968a41300128be35" Jul 10 19:08:14.375138 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.375116 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99"] Jul 10 19:08:14.378441 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.378415 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-33ca20/auth-test-model-33ca20-kserve-7d594f46b7-l4m99"] Jul 10 19:08:14.388751 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.388731 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e481ac0-f528-4f3d-92b5-13b3541b23bb-tls-certs\") pod \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " Jul 10 19:08:14.388834 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.388784 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-dshm\") pod \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " Jul 10 19:08:14.388923 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.388906 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-tmp-dir\") pod \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " Jul 10 19:08:14.388959 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.388936 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flbbt\" (UniqueName: \"kubernetes.io/projected/6e481ac0-f528-4f3d-92b5-13b3541b23bb-kube-api-access-flbbt\") pod \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " Jul 10 19:08:14.388996 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.388988 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-model-cache\") pod \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " Jul 10 19:08:14.389058 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.389043 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-home\") pod \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\" (UID: \"6e481ac0-f528-4f3d-92b5-13b3541b23bb\") " Jul 10 19:08:14.389377 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.389174 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e481ac0-f528-4f3d-92b5-13b3541b23bb" (UID: "6e481ac0-f528-4f3d-92b5-13b3541b23bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:08:14.389377 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.389312 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e481ac0-f528-4f3d-92b5-13b3541b23bb" (UID: "6e481ac0-f528-4f3d-92b5-13b3541b23bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:08:14.389377 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.389369 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-home" (OuterVolumeSpecName: "home") pod "6e481ac0-f528-4f3d-92b5-13b3541b23bb" (UID: "6e481ac0-f528-4f3d-92b5-13b3541b23bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:08:14.389557 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.389462 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-tmp-dir\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:08:14.389557 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.389481 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-model-cache\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:08:14.389557 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.389495 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-home\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:08:14.391492 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.391439 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-dshm" (OuterVolumeSpecName: "dshm") pod "6e481ac0-f528-4f3d-92b5-13b3541b23bb" (UID: "6e481ac0-f528-4f3d-92b5-13b3541b23bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:08:14.391492 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.391462 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e481ac0-f528-4f3d-92b5-13b3541b23bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e481ac0-f528-4f3d-92b5-13b3541b23bb" (UID: "6e481ac0-f528-4f3d-92b5-13b3541b23bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:08:14.391596 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.391521 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e481ac0-f528-4f3d-92b5-13b3541b23bb-kube-api-access-flbbt" (OuterVolumeSpecName: "kube-api-access-flbbt") pod "6e481ac0-f528-4f3d-92b5-13b3541b23bb" (UID: "6e481ac0-f528-4f3d-92b5-13b3541b23bb"). InnerVolumeSpecName "kube-api-access-flbbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:08:14.490366 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.490321 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e481ac0-f528-4f3d-92b5-13b3541b23bb-tls-certs\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:08:14.490366 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.490375 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e481ac0-f528-4f3d-92b5-13b3541b23bb-dshm\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:08:14.490570 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.490386 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flbbt\" (UniqueName: \"kubernetes.io/projected/6e481ac0-f528-4f3d-92b5-13b3541b23bb-kube-api-access-flbbt\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:08:14.674765 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.674734 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c"] Jul 10 19:08:14.678614 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:14.678587 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/auth-test-model-7b46a3-kserve-887bd6dbc-nfc8c"] Jul 10 19:08:15.844147 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:15.844111 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e481ac0-f528-4f3d-92b5-13b3541b23bb" path="/var/lib/kubelet/pods/6e481ac0-f528-4f3d-92b5-13b3541b23bb/volumes" Jul 10 19:08:15.844657 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:15.844641 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f41a8f1-9be4-439a-ab32-5ed564583d97" path="/var/lib/kubelet/pods/8f41a8f1-9be4-439a-ab32-5ed564583d97/volumes" Jul 10 19:08:25.500886 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.500848 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k"] Jul 10 19:08:25.501363 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.501338 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e481ac0-f528-4f3d-92b5-13b3541b23bb" containerName="main" Jul 10 19:08:25.501413 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.501369 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e481ac0-f528-4f3d-92b5-13b3541b23bb" containerName="main" Jul 10 19:08:25.501413 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.501389 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f41a8f1-9be4-439a-ab32-5ed564583d97" containerName="main" Jul 10 19:08:25.501413 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.501396 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f41a8f1-9be4-439a-ab32-5ed564583d97" containerName="main" Jul 10 19:08:25.501514 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.501502 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f41a8f1-9be4-439a-ab32-5ed564583d97" containerName="main" Jul 10 19:08:25.501548 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.501513 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e481ac0-f528-4f3d-92b5-13b3541b23bb" containerName="main" Jul 10 19:08:25.504699 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.504680 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.507233 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.507209 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-33ca20\"/\"kube-root-ca.crt\"" Jul 10 19:08:25.509063 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.508591 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-33ca20\"/\"openshift-service-ca.crt\"" Jul 10 19:08:25.509063 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.508623 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-33ca20\"/\"sub-test-model-33ca20-kserve-self-signed-certs\"" Jul 10 19:08:25.509063 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.508674 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-33ca20\"/\"default-dockercfg-wvv54\"" Jul 10 19:08:25.511368 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.511334 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k"] Jul 10 19:08:25.583844 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.583815 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-dshm\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.583844 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.583845 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-tls-certs\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.584044 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.583886 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-tmp-dir\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.584044 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.583910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qcm\" (UniqueName: \"kubernetes.io/projected/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-kube-api-access-k6qcm\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.584044 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.583939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-home\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.584044 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.583970 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-model-cache\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.684633 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.684588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-dshm\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.684633 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.684626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-tls-certs\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.684891 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.684679 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-tmp-dir\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.684891 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.684701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qcm\" (UniqueName: \"kubernetes.io/projected/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-kube-api-access-k6qcm\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.684891 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.684737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-home\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.684891 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.684773 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-model-cache\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.685519 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.685303 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-tmp-dir\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.685519 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.685326 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-home\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.685519 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.685525 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-model-cache\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.687486 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.687465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-dshm\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.688146 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.688123 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-tls-certs\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.693646 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.693620 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qcm\" (UniqueName: \"kubernetes.io/projected/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-kube-api-access-k6qcm\") pod \"sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.814601 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.814517 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:25.945901 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:25.945843 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k"] Jul 10 19:08:25.948721 ip-10-0-140-180 kubenswrapper[2576]: W0710 19:08:25.948694 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ac9e8dc_e6d9_405b_88f0_c67eeaf02a2c.slice/crio-dde4e0f8311c7617381beefe3e702b3de835ee1a16aba52035355c9a4252b015 WatchSource:0}: Error finding container dde4e0f8311c7617381beefe3e702b3de835ee1a16aba52035355c9a4252b015: Status 404 returned error can't find the container with id dde4e0f8311c7617381beefe3e702b3de835ee1a16aba52035355c9a4252b015 Jul 10 19:08:26.427231 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:26.427198 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" event={"ID":"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c","Type":"ContainerStarted","Data":"5523056aa791fbfc03d0c5ff019b4a3bb1ddb62dff957ecee2c4fbdff1da76cc"} Jul 10 19:08:26.427231 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:26.427237 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" event={"ID":"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c","Type":"ContainerStarted","Data":"dde4e0f8311c7617381beefe3e702b3de835ee1a16aba52035355c9a4252b015"} Jul 10 19:08:26.443971 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:26.443923 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" podStartSLOduration=1.4439083639999999 podStartE2EDuration="1.443908364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:26.442623797 +0000 UTC m=+2353.120336259" watchObservedRunningTime="2026-07-10 19:08:26.443908364 +0000 UTC m=+2353.121620826" Jul 10 19:08:29.689437 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:29.689405 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:08:35.815023 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:35.814990 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:35.815542 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:35.815125 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:35.827571 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:35.827550 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:36.489077 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:36.489042 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:08:55.496292 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.496254 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j"] Jul 10 19:08:55.499595 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.499575 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.502900 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.502875 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7b46a3\"/\"default-dockercfg-ql8fs\"" Jul 10 19:08:55.503035 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.502879 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7b46a3\"/\"kube-root-ca.crt\"" Jul 10 19:08:55.503035 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.502941 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7b46a3\"/\"sub-test-model-7b46a3-kserve-self-signed-certs\"" Jul 10 19:08:55.503035 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.502995 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7b46a3\"/\"openshift-service-ca.crt\"" Jul 10 19:08:55.508714 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.508693 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j"] Jul 10 19:08:55.648290 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.648253 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-home\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.648485 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.648296 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56b9r\" (UniqueName: \"kubernetes.io/projected/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-kube-api-access-56b9r\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.648485 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.648432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-model-cache\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.648485 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.648469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-dshm\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.648631 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.648489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-tmp-dir\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.648631 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.648539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-tls-certs\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.749777 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.749674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-home\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.749777 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.749718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56b9r\" (UniqueName: \"kubernetes.io/projected/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-kube-api-access-56b9r\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.749777 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.749771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-model-cache\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.750068 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.749794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-dshm\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.750068 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.749809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-tmp-dir\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.750068 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.749849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-tls-certs\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.750234 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.750184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-home\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.750234 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.750220 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-model-cache\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.750373 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.750302 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-tmp-dir\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.752234 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.752212 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-dshm\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.752385 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.752282 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-tls-certs\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.757068 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.757047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56b9r\" (UniqueName: \"kubernetes.io/projected/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-kube-api-access-56b9r\") pod \"sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.810136 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.810098 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:08:55.951945 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:55.951908 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j"] Jul 10 19:08:55.952921 ip-10-0-140-180 kubenswrapper[2576]: W0710 19:08:55.952896 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6ba6029_8895_4f5f_9fff_1e46eaaadf47.slice/crio-c554c46ba3777c44611fa3aa159e36eb53e73448c96007e2891b90621377acf1 WatchSource:0}: Error finding container c554c46ba3777c44611fa3aa159e36eb53e73448c96007e2891b90621377acf1: Status 404 returned error can't find the container with id c554c46ba3777c44611fa3aa159e36eb53e73448c96007e2891b90621377acf1 Jul 10 19:08:56.586334 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:56.586297 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" event={"ID":"b6ba6029-8895-4f5f-9fff-1e46eaaadf47","Type":"ContainerStarted","Data":"100f96aaac8381afcd98f37ea6371b46fa5dbc4f39deb7582201ba44a0a919ab"} Jul 10 19:08:56.586734 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:56.586345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" event={"ID":"b6ba6029-8895-4f5f-9fff-1e46eaaadf47","Type":"ContainerStarted","Data":"c554c46ba3777c44611fa3aa159e36eb53e73448c96007e2891b90621377acf1"} Jul 10 19:08:56.605597 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:08:56.605550 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" podStartSLOduration=1.605530844 podStartE2EDuration="1.605530844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:56.60375507 +0000 UTC m=+2383.281467559" watchObservedRunningTime="2026-07-10 19:08:56.605530844 +0000 UTC m=+2383.283243303" Jul 10 19:09:00.391410 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:00.391379 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:09:05.811090 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:05.811050 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:09:05.811614 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:05.811109 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:09:05.828778 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:05.828752 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:09:06.655927 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:06.655898 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:09:13.982765 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:13.982735 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 19:09:13.985707 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:13.985686 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzct8_bf1d25c8-e207-43d4-8a1d-6db4146f16eb/ovn-acl-logging/0.log" Jul 10 19:09:14.889109 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:14.889074 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:09:35.286684 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:35.286649 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:09:40.988392 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:40.988298 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k"] Jul 10 19:09:41.880938 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:41.880903 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j"] Jul 10 19:09:54.385941 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:54.385909 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:09:56.008529 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.008489 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" podUID="2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c" containerName="main" containerID="cri-o://5523056aa791fbfc03d0c5ff019b4a3bb1ddb62dff957ecee2c4fbdff1da76cc" gracePeriod=45 Jul 10 19:09:56.259472 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.259412 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:09:56.285060 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.285036 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-tls-certs\") pod \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " Jul 10 19:09:56.285195 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.285095 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6qcm\" (UniqueName: \"kubernetes.io/projected/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-kube-api-access-k6qcm\") pod \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " Jul 10 19:09:56.285195 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.285126 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-home\") pod \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " Jul 10 19:09:56.285195 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.285170 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-dshm\") pod \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " Jul 10 19:09:56.285195 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.285192 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-model-cache\") pod \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " Jul 10 19:09:56.285436 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.285239 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-tmp-dir\") pod \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\" (UID: \"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c\") " Jul 10 19:09:56.285499 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.285447 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-home" (OuterVolumeSpecName: "home") pod "2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c" (UID: "2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:09:56.285553 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.285494 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c" (UID: "2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:09:56.285685 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.285659 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c" (UID: "2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:09:56.287218 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.287190 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c" (UID: "2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:09:56.287595 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.287579 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-dshm" (OuterVolumeSpecName: "dshm") pod "2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c" (UID: "2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:09:56.287665 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.287617 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-kube-api-access-k6qcm" (OuterVolumeSpecName: "kube-api-access-k6qcm") pod "2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c" (UID: "2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c"). InnerVolumeSpecName "kube-api-access-k6qcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:09:56.386671 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.386643 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-tls-certs\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:09:56.386671 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.386669 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6qcm\" (UniqueName: \"kubernetes.io/projected/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-kube-api-access-k6qcm\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:09:56.386861 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.386681 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-home\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:09:56.386861 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.386692 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-dshm\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:09:56.386861 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.386705 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-model-cache\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:09:56.386861 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.386716 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c-tmp-dir\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:09:56.901114 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.901052 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" podUID="b6ba6029-8895-4f5f-9fff-1e46eaaadf47" containerName="main" containerID="cri-o://100f96aaac8381afcd98f37ea6371b46fa5dbc4f39deb7582201ba44a0a919ab" gracePeriod=45 Jul 10 19:09:56.907040 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.907010 2576 generic.go:356] "Generic (PLEG): container finished" podID="2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c" containerID="5523056aa791fbfc03d0c5ff019b4a3bb1ddb62dff957ecee2c4fbdff1da76cc" exitCode=0 Jul 10 19:09:56.907154 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.907052 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" Jul 10 19:09:56.907154 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.907064 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" event={"ID":"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c","Type":"ContainerDied","Data":"5523056aa791fbfc03d0c5ff019b4a3bb1ddb62dff957ecee2c4fbdff1da76cc"} Jul 10 19:09:56.907154 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.907101 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k" event={"ID":"2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c","Type":"ContainerDied","Data":"dde4e0f8311c7617381beefe3e702b3de835ee1a16aba52035355c9a4252b015"} Jul 10 19:09:56.907154 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.907122 2576 scope.go:117] "RemoveContainer" containerID="5523056aa791fbfc03d0c5ff019b4a3bb1ddb62dff957ecee2c4fbdff1da76cc" Jul 10 19:09:56.915752 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.915733 2576 scope.go:117] "RemoveContainer" containerID="5523056aa791fbfc03d0c5ff019b4a3bb1ddb62dff957ecee2c4fbdff1da76cc" Jul 10 19:09:56.915991 ip-10-0-140-180 kubenswrapper[2576]: E0710 19:09:56.915976 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5523056aa791fbfc03d0c5ff019b4a3bb1ddb62dff957ecee2c4fbdff1da76cc\": container with ID starting with 5523056aa791fbfc03d0c5ff019b4a3bb1ddb62dff957ecee2c4fbdff1da76cc not found: ID does not exist" containerID="5523056aa791fbfc03d0c5ff019b4a3bb1ddb62dff957ecee2c4fbdff1da76cc" Jul 10 19:09:56.916032 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.915998 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5523056aa791fbfc03d0c5ff019b4a3bb1ddb62dff957ecee2c4fbdff1da76cc"} err="failed to get container status \"5523056aa791fbfc03d0c5ff019b4a3bb1ddb62dff957ecee2c4fbdff1da76cc\": rpc error: code = NotFound desc = could not find container \"5523056aa791fbfc03d0c5ff019b4a3bb1ddb62dff957ecee2c4fbdff1da76cc\": container with ID starting with 5523056aa791fbfc03d0c5ff019b4a3bb1ddb62dff957ecee2c4fbdff1da76cc not found: ID does not exist" Jul 10 19:09:56.983035 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.983007 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k"] Jul 10 19:09:56.985065 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:56.985047 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-33ca20/sub-test-model-33ca20-kserve-5ffcd65c7f-s5z8k"] Jul 10 19:09:57.149603 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.149577 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:09:57.193568 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.193482 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56b9r\" (UniqueName: \"kubernetes.io/projected/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-kube-api-access-56b9r\") pod \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " Jul 10 19:09:57.193568 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.193530 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-dshm\") pod \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " Jul 10 19:09:57.193794 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.193621 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-model-cache\") pod \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " Jul 10 19:09:57.193794 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.193644 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-tmp-dir\") pod \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " Jul 10 19:09:57.193794 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.193689 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-home\") pod \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " Jul 10 19:09:57.193794 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.193713 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-tls-certs\") pod \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\" (UID: \"b6ba6029-8895-4f5f-9fff-1e46eaaadf47\") " Jul 10 19:09:57.193980 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.193866 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6ba6029-8895-4f5f-9fff-1e46eaaadf47" (UID: "b6ba6029-8895-4f5f-9fff-1e46eaaadf47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:09:57.193980 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.193947 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6ba6029-8895-4f5f-9fff-1e46eaaadf47" (UID: "b6ba6029-8895-4f5f-9fff-1e46eaaadf47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:09:57.193980 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.193964 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-home" (OuterVolumeSpecName: "home") pod "b6ba6029-8895-4f5f-9fff-1e46eaaadf47" (UID: "b6ba6029-8895-4f5f-9fff-1e46eaaadf47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:09:57.194139 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.194016 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-model-cache\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:09:57.195729 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.195700 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6ba6029-8895-4f5f-9fff-1e46eaaadf47" (UID: "b6ba6029-8895-4f5f-9fff-1e46eaaadf47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:09:57.195729 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.195715 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-kube-api-access-56b9r" (OuterVolumeSpecName: "kube-api-access-56b9r") pod "b6ba6029-8895-4f5f-9fff-1e46eaaadf47" (UID: "b6ba6029-8895-4f5f-9fff-1e46eaaadf47"). InnerVolumeSpecName "kube-api-access-56b9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:09:57.195929 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.195756 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-dshm" (OuterVolumeSpecName: "dshm") pod "b6ba6029-8895-4f5f-9fff-1e46eaaadf47" (UID: "b6ba6029-8895-4f5f-9fff-1e46eaaadf47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:09:57.295229 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.295195 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-tmp-dir\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:09:57.295229 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.295224 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-home\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:09:57.295229 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.295236 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-tls-certs\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:09:57.295514 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.295247 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56b9r\" (UniqueName: \"kubernetes.io/projected/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-kube-api-access-56b9r\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:09:57.295514 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.295258 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6ba6029-8895-4f5f-9fff-1e46eaaadf47-dshm\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:09:57.844017 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.843983 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c" path="/var/lib/kubelet/pods/2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c/volumes" Jul 10 19:09:57.913070 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.913035 2576 generic.go:356] "Generic (PLEG): container finished" podID="b6ba6029-8895-4f5f-9fff-1e46eaaadf47" containerID="100f96aaac8381afcd98f37ea6371b46fa5dbc4f39deb7582201ba44a0a919ab" exitCode=0 Jul 10 19:09:57.913239 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.913102 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" Jul 10 19:09:57.913239 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.913114 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" event={"ID":"b6ba6029-8895-4f5f-9fff-1e46eaaadf47","Type":"ContainerDied","Data":"100f96aaac8381afcd98f37ea6371b46fa5dbc4f39deb7582201ba44a0a919ab"} Jul 10 19:09:57.913239 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.913157 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j" event={"ID":"b6ba6029-8895-4f5f-9fff-1e46eaaadf47","Type":"ContainerDied","Data":"c554c46ba3777c44611fa3aa159e36eb53e73448c96007e2891b90621377acf1"} Jul 10 19:09:57.913239 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.913180 2576 scope.go:117] "RemoveContainer" containerID="100f96aaac8381afcd98f37ea6371b46fa5dbc4f39deb7582201ba44a0a919ab" Jul 10 19:09:57.921978 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.921962 2576 scope.go:117] "RemoveContainer" containerID="100f96aaac8381afcd98f37ea6371b46fa5dbc4f39deb7582201ba44a0a919ab" Jul 10 19:09:57.922254 ip-10-0-140-180 kubenswrapper[2576]: E0710 19:09:57.922237 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"100f96aaac8381afcd98f37ea6371b46fa5dbc4f39deb7582201ba44a0a919ab\": container with ID starting with 100f96aaac8381afcd98f37ea6371b46fa5dbc4f39deb7582201ba44a0a919ab not found: ID does not exist" containerID="100f96aaac8381afcd98f37ea6371b46fa5dbc4f39deb7582201ba44a0a919ab" Jul 10 19:09:57.922306 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.922261 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"100f96aaac8381afcd98f37ea6371b46fa5dbc4f39deb7582201ba44a0a919ab"} err="failed to get container status \"100f96aaac8381afcd98f37ea6371b46fa5dbc4f39deb7582201ba44a0a919ab\": rpc error: code = NotFound desc = could not find container \"100f96aaac8381afcd98f37ea6371b46fa5dbc4f39deb7582201ba44a0a919ab\": container with ID starting with 100f96aaac8381afcd98f37ea6371b46fa5dbc4f39deb7582201ba44a0a919ab not found: ID does not exist" Jul 10 19:09:57.929594 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.929571 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j"] Jul 10 19:09:57.934910 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:57.934888 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/sub-test-model-7b46a3-kserve-6544bfb8b5-qfd8j"] Jul 10 19:09:59.843971 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:09:59.843937 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6ba6029-8895-4f5f-9fff-1e46eaaadf47" path="/var/lib/kubelet/pods/b6ba6029-8895-4f5f-9fff-1e46eaaadf47/volumes" Jul 10 19:10:07.286892 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:07.286850 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:10:27.591726 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.591695 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s"] Jul 10 19:10:27.592157 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.592050 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6ba6029-8895-4f5f-9fff-1e46eaaadf47" containerName="main" Jul 10 19:10:27.592157 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.592062 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6ba6029-8895-4f5f-9fff-1e46eaaadf47" containerName="main" Jul 10 19:10:27.592157 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.592081 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c" containerName="main" Jul 10 19:10:27.592157 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.592090 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c" containerName="main" Jul 10 19:10:27.592157 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.592151 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6ba6029-8895-4f5f-9fff-1e46eaaadf47" containerName="main" Jul 10 19:10:27.592402 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.592163 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac9e8dc-e6d9-405b-88f0-c67eeaf02a2c" containerName="main" Jul 10 19:10:27.594959 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.594944 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.598331 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.598307 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7b46a3\"/\"kube-root-ca.crt\"" Jul 10 19:10:27.598563 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.598314 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7b46a3\"/\"rate-test-model-7b46a3-kserve-self-signed-certs\"" Jul 10 19:10:27.598701 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.598326 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7b46a3\"/\"openshift-service-ca.crt\"" Jul 10 19:10:27.598701 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.598418 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7b46a3\"/\"default-dockercfg-ql8fs\"" Jul 10 19:10:27.605560 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.605537 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s"] Jul 10 19:10:27.651403 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.651370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-model-cache\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.651561 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.651409 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-dshm\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.651561 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.651444 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv8qj\" (UniqueName: \"kubernetes.io/projected/595fc6ae-933a-4e0e-817f-afda1a40cc07-kube-api-access-fv8qj\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.651561 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.651521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595fc6ae-933a-4e0e-817f-afda1a40cc07-tls-certs\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.651699 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.651564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-tmp-dir\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.651699 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.651603 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-home\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.752788 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.752752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-home\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.752985 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.752808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-model-cache\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.752985 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.752829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-dshm\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.752985 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.752867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv8qj\" (UniqueName: \"kubernetes.io/projected/595fc6ae-933a-4e0e-817f-afda1a40cc07-kube-api-access-fv8qj\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.752985 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.752890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595fc6ae-933a-4e0e-817f-afda1a40cc07-tls-certs\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.752985 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.752922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-tmp-dir\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.753292 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.753264 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-model-cache\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.753373 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.753294 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-home\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.753373 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.753320 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-tmp-dir\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.755177 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.755149 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-dshm\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.755375 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.755336 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595fc6ae-933a-4e0e-817f-afda1a40cc07-tls-certs\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.760840 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.760817 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv8qj\" (UniqueName: \"kubernetes.io/projected/595fc6ae-933a-4e0e-817f-afda1a40cc07-kube-api-access-fv8qj\") pod \"rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:27.905538 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:27.905451 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:28.037250 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:28.037225 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s"] Jul 10 19:10:28.039284 ip-10-0-140-180 kubenswrapper[2576]: W0710 19:10:28.039255 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod595fc6ae_933a_4e0e_817f_afda1a40cc07.slice/crio-c8dbdad14c7c77af9dae8c9484d7045374d7bd78da42d9dfb0a5bca980d6e6bb WatchSource:0}: Error finding container c8dbdad14c7c77af9dae8c9484d7045374d7bd78da42d9dfb0a5bca980d6e6bb: Status 404 returned error can't find the container with id c8dbdad14c7c77af9dae8c9484d7045374d7bd78da42d9dfb0a5bca980d6e6bb Jul 10 19:10:28.074295 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:28.074269 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" event={"ID":"595fc6ae-933a-4e0e-817f-afda1a40cc07","Type":"ContainerStarted","Data":"c8dbdad14c7c77af9dae8c9484d7045374d7bd78da42d9dfb0a5bca980d6e6bb"} Jul 10 19:10:29.084686 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:29.084634 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" event={"ID":"595fc6ae-933a-4e0e-817f-afda1a40cc07","Type":"ContainerStarted","Data":"ac99d157b029012f9ee2e09ad11281fd825f6884115e7ad982057ff0792f89f2"} Jul 10 19:10:29.103310 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:29.103261 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" podStartSLOduration=2.103247595 podStartE2EDuration="2.103247595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:29.101604688 +0000 UTC m=+2475.779317151" watchObservedRunningTime="2026-07-10 19:10:29.103247595 +0000 UTC m=+2475.780960057" Jul 10 19:10:32.389028 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:32.388994 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:10:37.906609 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:37.906566 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:37.906609 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:37.906620 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:37.919228 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:37.919202 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:38.143235 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:38.143208 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:10:47.589337 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:10:47.589293 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:11:03.590116 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:03.590081 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:11:10.972982 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:10.972949 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s"] Jul 10 19:11:18.190888 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:18.190852 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:11:25.994812 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:25.994749 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" podUID="595fc6ae-933a-4e0e-817f-afda1a40cc07" containerName="main" containerID="cri-o://ac99d157b029012f9ee2e09ad11281fd825f6884115e7ad982057ff0792f89f2" gracePeriod=45 Jul 10 19:11:26.239804 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.239777 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:11:26.362635 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.362598 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv8qj\" (UniqueName: \"kubernetes.io/projected/595fc6ae-933a-4e0e-817f-afda1a40cc07-kube-api-access-fv8qj\") pod \"595fc6ae-933a-4e0e-817f-afda1a40cc07\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " Jul 10 19:11:26.362832 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.362664 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-model-cache\") pod \"595fc6ae-933a-4e0e-817f-afda1a40cc07\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " Jul 10 19:11:26.362832 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.362733 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595fc6ae-933a-4e0e-817f-afda1a40cc07-tls-certs\") pod \"595fc6ae-933a-4e0e-817f-afda1a40cc07\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " Jul 10 19:11:26.362832 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.362773 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-home\") pod \"595fc6ae-933a-4e0e-817f-afda1a40cc07\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " Jul 10 19:11:26.362832 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.362821 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-tmp-dir\") pod \"595fc6ae-933a-4e0e-817f-afda1a40cc07\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " Jul 10 19:11:26.363043 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.362849 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-dshm\") pod \"595fc6ae-933a-4e0e-817f-afda1a40cc07\" (UID: \"595fc6ae-933a-4e0e-817f-afda1a40cc07\") " Jul 10 19:11:26.363043 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.362974 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-model-cache" (OuterVolumeSpecName: "model-cache") pod "595fc6ae-933a-4e0e-817f-afda1a40cc07" (UID: "595fc6ae-933a-4e0e-817f-afda1a40cc07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:11:26.363043 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.363012 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-home" (OuterVolumeSpecName: "home") pod "595fc6ae-933a-4e0e-817f-afda1a40cc07" (UID: "595fc6ae-933a-4e0e-817f-afda1a40cc07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:11:26.363150 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.363113 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "595fc6ae-933a-4e0e-817f-afda1a40cc07" (UID: "595fc6ae-933a-4e0e-817f-afda1a40cc07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:11:26.363191 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.363165 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-model-cache\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:11:26.363191 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.363185 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-home\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:11:26.364817 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.364791 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595fc6ae-933a-4e0e-817f-afda1a40cc07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "595fc6ae-933a-4e0e-817f-afda1a40cc07" (UID: "595fc6ae-933a-4e0e-817f-afda1a40cc07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:11:26.364977 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.364872 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595fc6ae-933a-4e0e-817f-afda1a40cc07-kube-api-access-fv8qj" (OuterVolumeSpecName: "kube-api-access-fv8qj") pod "595fc6ae-933a-4e0e-817f-afda1a40cc07" (UID: "595fc6ae-933a-4e0e-817f-afda1a40cc07"). InnerVolumeSpecName "kube-api-access-fv8qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:11:26.364977 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.364906 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-dshm" (OuterVolumeSpecName: "dshm") pod "595fc6ae-933a-4e0e-817f-afda1a40cc07" (UID: "595fc6ae-933a-4e0e-817f-afda1a40cc07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:11:26.383969 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.383938 2576 generic.go:356] "Generic (PLEG): container finished" podID="595fc6ae-933a-4e0e-817f-afda1a40cc07" containerID="ac99d157b029012f9ee2e09ad11281fd825f6884115e7ad982057ff0792f89f2" exitCode=0 Jul 10 19:11:26.384073 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.383979 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" Jul 10 19:11:26.384073 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.383976 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" event={"ID":"595fc6ae-933a-4e0e-817f-afda1a40cc07","Type":"ContainerDied","Data":"ac99d157b029012f9ee2e09ad11281fd825f6884115e7ad982057ff0792f89f2"} Jul 10 19:11:26.384164 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.384100 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s" event={"ID":"595fc6ae-933a-4e0e-817f-afda1a40cc07","Type":"ContainerDied","Data":"c8dbdad14c7c77af9dae8c9484d7045374d7bd78da42d9dfb0a5bca980d6e6bb"} Jul 10 19:11:26.385118 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.384121 2576 scope.go:117] "RemoveContainer" containerID="ac99d157b029012f9ee2e09ad11281fd825f6884115e7ad982057ff0792f89f2" Jul 10 19:11:26.394436 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.394413 2576 scope.go:117] "RemoveContainer" containerID="ac99d157b029012f9ee2e09ad11281fd825f6884115e7ad982057ff0792f89f2" Jul 10 19:11:26.394681 ip-10-0-140-180 kubenswrapper[2576]: E0710 19:11:26.394665 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac99d157b029012f9ee2e09ad11281fd825f6884115e7ad982057ff0792f89f2\": container with ID starting with ac99d157b029012f9ee2e09ad11281fd825f6884115e7ad982057ff0792f89f2 not found: ID does not exist" containerID="ac99d157b029012f9ee2e09ad11281fd825f6884115e7ad982057ff0792f89f2" Jul 10 19:11:26.394742 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.394686 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac99d157b029012f9ee2e09ad11281fd825f6884115e7ad982057ff0792f89f2"} err="failed to get container status \"ac99d157b029012f9ee2e09ad11281fd825f6884115e7ad982057ff0792f89f2\": rpc error: code = NotFound desc = could not find container \"ac99d157b029012f9ee2e09ad11281fd825f6884115e7ad982057ff0792f89f2\": container with ID starting with ac99d157b029012f9ee2e09ad11281fd825f6884115e7ad982057ff0792f89f2 not found: ID does not exist" Jul 10 19:11:26.405890 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.405868 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s"] Jul 10 19:11:26.413772 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.413752 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7b46a3/rate-test-model-7b46a3-kserve-858b86dfd7-9ls8s"] Jul 10 19:11:26.463655 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.463620 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv8qj\" (UniqueName: \"kubernetes.io/projected/595fc6ae-933a-4e0e-817f-afda1a40cc07-kube-api-access-fv8qj\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:11:26.463655 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.463647 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595fc6ae-933a-4e0e-817f-afda1a40cc07-tls-certs\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:11:26.463655 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.463658 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-tmp-dir\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:11:26.463884 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.463667 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595fc6ae-933a-4e0e-817f-afda1a40cc07-dshm\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:11:26.857428 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.857393 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7367-139972880132976"] Jul 10 19:11:26.857737 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.857726 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="595fc6ae-933a-4e0e-817f-afda1a40cc07" containerName="main" Jul 10 19:11:26.857785 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.857739 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="595fc6ae-933a-4e0e-817f-afda1a40cc07" containerName="main" Jul 10 19:11:26.857823 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.857796 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="595fc6ae-933a-4e0e-817f-afda1a40cc07" containerName="main" Jul 10 19:11:26.861836 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.861819 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7367-139972880132976" Jul 10 19:11:26.864260 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.864236 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-jhhrg\"" Jul 10 19:11:26.866982 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.866958 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7367-139972880132976"] Jul 10 19:11:26.969046 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:26.969006 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b56kq\" (UniqueName: \"kubernetes.io/projected/0b02fefa-093b-4f12-ad85-f16f885e448b-kube-api-access-b56kq\") pod \"test-curl-7367-139972880132976\" (UID: \"0b02fefa-093b-4f12-ad85-f16f885e448b\") " pod="openshift-ingress/test-curl-7367-139972880132976" Jul 10 19:11:27.069470 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:27.069434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b56kq\" (UniqueName: \"kubernetes.io/projected/0b02fefa-093b-4f12-ad85-f16f885e448b-kube-api-access-b56kq\") pod \"test-curl-7367-139972880132976\" (UID: \"0b02fefa-093b-4f12-ad85-f16f885e448b\") " pod="openshift-ingress/test-curl-7367-139972880132976" Jul 10 19:11:27.077200 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:27.077168 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b56kq\" (UniqueName: \"kubernetes.io/projected/0b02fefa-093b-4f12-ad85-f16f885e448b-kube-api-access-b56kq\") pod \"test-curl-7367-139972880132976\" (UID: \"0b02fefa-093b-4f12-ad85-f16f885e448b\") " pod="openshift-ingress/test-curl-7367-139972880132976" Jul 10 19:11:27.175596 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:27.175522 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7367-139972880132976" Jul 10 19:11:27.302681 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:27.302654 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7367-139972880132976"] Jul 10 19:11:27.304830 ip-10-0-140-180 kubenswrapper[2576]: W0710 19:11:27.304800 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b02fefa_093b_4f12_ad85_f16f885e448b.slice/crio-596c69f5951cdb10dd2253ff678fbb02bca39af550a2313b0ba20f32c9cda786 WatchSource:0}: Error finding container 596c69f5951cdb10dd2253ff678fbb02bca39af550a2313b0ba20f32c9cda786: Status 404 returned error can't find the container with id 596c69f5951cdb10dd2253ff678fbb02bca39af550a2313b0ba20f32c9cda786 Jul 10 19:11:27.306139 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:27.306118 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:11:27.389893 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:27.389862 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7367-139972880132976" event={"ID":"0b02fefa-093b-4f12-ad85-f16f885e448b","Type":"ContainerStarted","Data":"596c69f5951cdb10dd2253ff678fbb02bca39af550a2313b0ba20f32c9cda786"} Jul 10 19:11:27.844037 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:27.844003 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="595fc6ae-933a-4e0e-817f-afda1a40cc07" path="/var/lib/kubelet/pods/595fc6ae-933a-4e0e-817f-afda1a40cc07/volumes" Jul 10 19:11:28.397591 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:28.397503 2576 generic.go:356] "Generic (PLEG): container finished" podID="0b02fefa-093b-4f12-ad85-f16f885e448b" containerID="f365676545117c61ebdef772668143320ae012fd14e6afc552ded1a281401ab9" exitCode=0 Jul 10 19:11:28.397591 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:28.397545 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7367-139972880132976" event={"ID":"0b02fefa-093b-4f12-ad85-f16f885e448b","Type":"ContainerDied","Data":"f365676545117c61ebdef772668143320ae012fd14e6afc552ded1a281401ab9"} Jul 10 19:11:29.540231 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:29.540207 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7367-139972880132976" Jul 10 19:11:29.573559 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:29.573536 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7367-139972880132976_0b02fefa-093b-4f12-ad85-f16f885e448b/test-curl-7367-139972880132976/0.log" Jul 10 19:11:29.596396 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:29.596332 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7367-139972880132976"] Jul 10 19:11:29.598383 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:29.598345 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7367-139972880132976"] Jul 10 19:11:29.691045 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:29.690963 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b56kq\" (UniqueName: \"kubernetes.io/projected/0b02fefa-093b-4f12-ad85-f16f885e448b-kube-api-access-b56kq\") pod \"0b02fefa-093b-4f12-ad85-f16f885e448b\" (UID: \"0b02fefa-093b-4f12-ad85-f16f885e448b\") " Jul 10 19:11:29.693039 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:29.693006 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b02fefa-093b-4f12-ad85-f16f885e448b-kube-api-access-b56kq" (OuterVolumeSpecName: "kube-api-access-b56kq") pod "0b02fefa-093b-4f12-ad85-f16f885e448b" (UID: "0b02fefa-093b-4f12-ad85-f16f885e448b"). InnerVolumeSpecName "kube-api-access-b56kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:11:29.791652 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:29.791620 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b56kq\" (UniqueName: \"kubernetes.io/projected/0b02fefa-093b-4f12-ad85-f16f885e448b-kube-api-access-b56kq\") on node \"ip-10-0-140-180.ec2.internal\" DevicePath \"\"" Jul 10 19:11:29.843421 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:29.843392 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b02fefa-093b-4f12-ad85-f16f885e448b" path="/var/lib/kubelet/pods/0b02fefa-093b-4f12-ad85-f16f885e448b/volumes" Jul 10 19:11:30.408116 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:30.408085 2576 scope.go:117] "RemoveContainer" containerID="f365676545117c61ebdef772668143320ae012fd14e6afc552ded1a281401ab9" Jul 10 19:11:30.408116 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:30.408106 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7367-139972880132976" Jul 10 19:11:34.391662 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:34.391627 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:11:46.786047 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:46.786011 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-hnt6h"] Jul 10 19:11:48.528982 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:48.528949 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-hfj7g_9600ebbe-f0bc-4265-884b-274efae32892/manager/0.log" Jul 10 19:11:48.651615 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:48.651591 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wkrqc_79685dda-acd4-4446-93b1-2608b69dd463/manager/0.log" Jul 10 19:11:48.873729 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:48.873704 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zjr4g_6dfdbad6-5aa2-4443-9398-3a59c736a589/server/0.log" Jul 10 19:11:48.990200 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:48.990168 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-czvzp_3be6a366-f512-4c82-887c-195e16cd0ea1/manager/0.log" Jul 10 19:11:49.336343 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:49.336315 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz_3bfd1a11-9e2b-4017-8996-699a64b4e0de/copy-manifests/0.log" Jul 10 19:11:49.355917 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:49.355891 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz_3bfd1a11-9e2b-4017-8996-699a64b4e0de/manager/0.log" Jul 10 19:11:50.509668 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:50.509639 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd_967cc74b-9c9f-4cd9-94aa-aa7199f74236/util/0.log" Jul 10 19:11:50.516667 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:50.516639 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd_967cc74b-9c9f-4cd9-94aa-aa7199f74236/pull/0.log" Jul 10 19:11:50.522778 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:50.522756 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd_967cc74b-9c9f-4cd9-94aa-aa7199f74236/extract/0.log" Jul 10 19:11:50.628339 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:50.628306 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6_96002491-e88e-4839-9c4d-7801d0c622d4/util/0.log" Jul 10 19:11:50.635660 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:50.635630 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6_96002491-e88e-4839-9c4d-7801d0c622d4/pull/0.log" Jul 10 19:11:50.641656 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:50.641633 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6_96002491-e88e-4839-9c4d-7801d0c622d4/extract/0.log" Jul 10 19:11:50.749308 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:50.749280 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz_b6f1c0a6-2710-4d70-9ac0-e34c95f25bac/util/0.log" Jul 10 19:11:50.755545 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:50.755510 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz_b6f1c0a6-2710-4d70-9ac0-e34c95f25bac/pull/0.log" Jul 10 19:11:50.761367 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:50.761284 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz_b6f1c0a6-2710-4d70-9ac0-e34c95f25bac/extract/0.log" Jul 10 19:11:50.868546 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:50.868513 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr_720edcea-b029-45a0-b300-9eac320b1d73/util/0.log" Jul 10 19:11:50.874983 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:50.874958 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr_720edcea-b029-45a0-b300-9eac320b1d73/pull/0.log" Jul 10 19:11:50.881220 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:50.881200 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr_720edcea-b029-45a0-b300-9eac320b1d73/extract/0.log" Jul 10 19:11:51.104463 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:51.104437 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-nmtvc_06cfb935-4eed-451d-a62e-c44fb9174a7c/manager/0.log" Jul 10 19:11:51.217840 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:51.217813 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-2bpss_d3170799-2ee0-4649-b100-efef023e67e6/manager/0.log" Jul 10 19:11:51.322599 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:51.322572 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-6zw5r_fe032d86-7163-4e5b-9403-0b1804b79c1a/kuadrant-console-plugin/0.log" Jul 10 19:11:51.442705 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:51.442628 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-tbc99_353eda98-7872-455e-9f82-e36f7df6d1c3/registry-server/0.log" Jul 10 19:11:51.561831 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:51.561803 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-k425n_28d2da83-2006-4378-bd15-4194b7b2b13f/manager/0.log" Jul 10 19:11:51.672663 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:51.672634 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-hnt6h_a2859ebf-9ede-4d35-bf81-34d56d12c5c7/limitador/0.log" Jul 10 19:11:52.135228 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:52.135180 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6787xd5g_e2af4c51-c115-4f25-8868-4ab579f9633a/istio-proxy/0.log" Jul 10 19:11:52.365654 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:52.365619 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d88546674-cpbkv_7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1/kube-auth-proxy/0.log" Jul 10 19:11:52.594922 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:52.594887 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-npkgq_f3d5f957-c477-465a-8199-48833698b766/istio-proxy/1.log" Jul 10 19:11:53.707463 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:53.707428 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-6lrds_28051884-0ba2-40a4-82c9-679d77da1252/main/0.log" Jul 10 19:11:53.810544 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:11:53.810511 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-8hwtx_2a96fd99-f457-4315-a7bb-5dc0bdc584b7/main/0.log" Jul 10 19:12:00.180756 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:00.180719 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rs2hc_ec189118-e07f-4ba5-a5c0-cc1910849ca6/global-pull-secret-syncer/0.log" Jul 10 19:12:00.305158 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:00.305129 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g54w6_19914451-ba36-4c62-a151-f74fc24b9102/konnectivity-agent/0.log" Jul 10 19:12:00.384588 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:00.384563 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-180.ec2.internal_34ce888784a2bd4fe2722ae68893f984/haproxy/0.log" Jul 10 19:12:04.867976 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:04.867933 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd_967cc74b-9c9f-4cd9-94aa-aa7199f74236/extract/0.log" Jul 10 19:12:04.889043 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:04.889014 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd_967cc74b-9c9f-4cd9-94aa-aa7199f74236/util/0.log" Jul 10 19:12:04.932430 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:04.932403 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75956qdd_967cc74b-9c9f-4cd9-94aa-aa7199f74236/pull/0.log" Jul 10 19:12:04.983771 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:04.983745 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6_96002491-e88e-4839-9c4d-7801d0c622d4/extract/0.log" Jul 10 19:12:05.004627 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:05.004599 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6_96002491-e88e-4839-9c4d-7801d0c622d4/util/0.log" Jul 10 19:12:05.026298 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:05.026276 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07klh6_96002491-e88e-4839-9c4d-7801d0c622d4/pull/0.log" Jul 10 19:12:05.052469 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:05.052445 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz_b6f1c0a6-2710-4d70-9ac0-e34c95f25bac/extract/0.log" Jul 10 19:12:05.072940 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:05.072921 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz_b6f1c0a6-2710-4d70-9ac0-e34c95f25bac/util/0.log" Jul 10 19:12:05.095783 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:05.095758 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kkddz_b6f1c0a6-2710-4d70-9ac0-e34c95f25bac/pull/0.log" Jul 10 19:12:05.125265 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:05.125176 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr_720edcea-b029-45a0-b300-9eac320b1d73/extract/0.log" Jul 10 19:12:05.151091 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:05.151066 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr_720edcea-b029-45a0-b300-9eac320b1d73/util/0.log" Jul 10 19:12:05.172278 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:05.172248 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1m69nr_720edcea-b029-45a0-b300-9eac320b1d73/pull/0.log" Jul 10 19:12:05.642304 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:05.642274 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-nmtvc_06cfb935-4eed-451d-a62e-c44fb9174a7c/manager/0.log" Jul 10 19:12:05.665616 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:05.665587 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-2bpss_d3170799-2ee0-4649-b100-efef023e67e6/manager/0.log" Jul 10 19:12:05.689711 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:05.689692 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-6zw5r_fe032d86-7163-4e5b-9403-0b1804b79c1a/kuadrant-console-plugin/0.log" Jul 10 19:12:05.738208 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:05.738179 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-tbc99_353eda98-7872-455e-9f82-e36f7df6d1c3/registry-server/0.log" Jul 10 19:12:05.906989 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:05.906902 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-k425n_28d2da83-2006-4378-bd15-4194b7b2b13f/manager/0.log" Jul 10 19:12:05.929342 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:05.929315 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-hnt6h_a2859ebf-9ede-4d35-bf81-34d56d12c5c7/limitador/0.log" Jul 10 19:12:08.197164 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:08.197123 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m26lv_d51265c6-f3cf-4982-85a9-327c4db64d6f/node-exporter/0.log" Jul 10 19:12:08.217982 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:08.217956 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m26lv_d51265c6-f3cf-4982-85a9-327c4db64d6f/kube-rbac-proxy/0.log" Jul 10 19:12:08.240738 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:08.240716 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m26lv_d51265c6-f3cf-4982-85a9-327c4db64d6f/init-textfile/0.log" Jul 10 19:12:09.965627 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:09.965593 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn"] Jul 10 19:12:09.966021 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:09.965976 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b02fefa-093b-4f12-ad85-f16f885e448b" containerName="test-curl-7367-139972880132976" Jul 10 19:12:09.966021 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:09.965992 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b02fefa-093b-4f12-ad85-f16f885e448b" containerName="test-curl-7367-139972880132976" Jul 10 19:12:09.966095 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:09.966064 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b02fefa-093b-4f12-ad85-f16f885e448b" containerName="test-curl-7367-139972880132976" Jul 10 19:12:09.971541 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:09.971523 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:09.974853 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:09.974828 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w9pq4\"/\"openshift-service-ca.crt\"" Jul 10 19:12:09.974980 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:09.974828 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w9pq4\"/\"default-dockercfg-spgnv\"" Jul 10 19:12:09.974980 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:09.974876 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w9pq4\"/\"kube-root-ca.crt\"" Jul 10 19:12:09.981037 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:09.981015 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn"] Jul 10 19:12:10.037909 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.037880 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/76f70bd6-e869-45dc-8aa4-7ed9b42fef75-podres\") pod \"perf-node-gather-daemonset-v8zjn\" (UID: \"76f70bd6-e869-45dc-8aa4-7ed9b42fef75\") " pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.038059 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.037912 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9mzr\" (UniqueName: \"kubernetes.io/projected/76f70bd6-e869-45dc-8aa4-7ed9b42fef75-kube-api-access-b9mzr\") pod \"perf-node-gather-daemonset-v8zjn\" (UID: \"76f70bd6-e869-45dc-8aa4-7ed9b42fef75\") " pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.038059 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.037993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76f70bd6-e869-45dc-8aa4-7ed9b42fef75-sys\") pod \"perf-node-gather-daemonset-v8zjn\" (UID: \"76f70bd6-e869-45dc-8aa4-7ed9b42fef75\") " pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.038059 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.038009 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76f70bd6-e869-45dc-8aa4-7ed9b42fef75-lib-modules\") pod \"perf-node-gather-daemonset-v8zjn\" (UID: \"76f70bd6-e869-45dc-8aa4-7ed9b42fef75\") " pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.038218 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.038087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/76f70bd6-e869-45dc-8aa4-7ed9b42fef75-proc\") pod \"perf-node-gather-daemonset-v8zjn\" (UID: \"76f70bd6-e869-45dc-8aa4-7ed9b42fef75\") " pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.139550 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.139515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76f70bd6-e869-45dc-8aa4-7ed9b42fef75-sys\") pod \"perf-node-gather-daemonset-v8zjn\" (UID: \"76f70bd6-e869-45dc-8aa4-7ed9b42fef75\") " pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.139550 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.139547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76f70bd6-e869-45dc-8aa4-7ed9b42fef75-lib-modules\") pod \"perf-node-gather-daemonset-v8zjn\" (UID: \"76f70bd6-e869-45dc-8aa4-7ed9b42fef75\") " pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.139727 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.139567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/76f70bd6-e869-45dc-8aa4-7ed9b42fef75-proc\") pod \"perf-node-gather-daemonset-v8zjn\" (UID: \"76f70bd6-e869-45dc-8aa4-7ed9b42fef75\") " pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.139727 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.139604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/76f70bd6-e869-45dc-8aa4-7ed9b42fef75-podres\") pod \"perf-node-gather-daemonset-v8zjn\" (UID: \"76f70bd6-e869-45dc-8aa4-7ed9b42fef75\") " pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.139727 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.139627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9mzr\" (UniqueName: \"kubernetes.io/projected/76f70bd6-e869-45dc-8aa4-7ed9b42fef75-kube-api-access-b9mzr\") pod \"perf-node-gather-daemonset-v8zjn\" (UID: \"76f70bd6-e869-45dc-8aa4-7ed9b42fef75\") " pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.139727 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.139638 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76f70bd6-e869-45dc-8aa4-7ed9b42fef75-sys\") pod \"perf-node-gather-daemonset-v8zjn\" (UID: \"76f70bd6-e869-45dc-8aa4-7ed9b42fef75\") " pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.139727 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.139667 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/76f70bd6-e869-45dc-8aa4-7ed9b42fef75-proc\") pod \"perf-node-gather-daemonset-v8zjn\" (UID: \"76f70bd6-e869-45dc-8aa4-7ed9b42fef75\") " pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.139727 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.139712 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/76f70bd6-e869-45dc-8aa4-7ed9b42fef75-podres\") pod \"perf-node-gather-daemonset-v8zjn\" (UID: \"76f70bd6-e869-45dc-8aa4-7ed9b42fef75\") " pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.139727 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.139713 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76f70bd6-e869-45dc-8aa4-7ed9b42fef75-lib-modules\") pod \"perf-node-gather-daemonset-v8zjn\" (UID: \"76f70bd6-e869-45dc-8aa4-7ed9b42fef75\") " pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.148559 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.148537 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9mzr\" (UniqueName: \"kubernetes.io/projected/76f70bd6-e869-45dc-8aa4-7ed9b42fef75-kube-api-access-b9mzr\") pod \"perf-node-gather-daemonset-v8zjn\" (UID: \"76f70bd6-e869-45dc-8aa4-7ed9b42fef75\") " pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.281911 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.281830 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.412946 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.412916 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn"] Jul 10 19:12:10.413290 ip-10-0-140-180 kubenswrapper[2576]: W0710 19:12:10.413257 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod76f70bd6_e869_45dc_8aa4_7ed9b42fef75.slice/crio-050958c9666259bf64872419fe0b57d7db60d1835b16ed6702496ac7706fb810 WatchSource:0}: Error finding container 050958c9666259bf64872419fe0b57d7db60d1835b16ed6702496ac7706fb810: Status 404 returned error can't find the container with id 050958c9666259bf64872419fe0b57d7db60d1835b16ed6702496ac7706fb810 Jul 10 19:12:10.621168 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.621130 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" event={"ID":"76f70bd6-e869-45dc-8aa4-7ed9b42fef75","Type":"ContainerStarted","Data":"431d578c623ad781984e4fd543b3f7115627164a3f91d1307e7d7cae2ee8d48c"} Jul 10 19:12:10.621337 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.621186 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" event={"ID":"76f70bd6-e869-45dc-8aa4-7ed9b42fef75","Type":"ContainerStarted","Data":"050958c9666259bf64872419fe0b57d7db60d1835b16ed6702496ac7706fb810"} Jul 10 19:12:10.621337 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.621206 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:10.638380 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:10.638313 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" podStartSLOduration=1.638294797 podStartE2EDuration="1.638294797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:12:10.637150293 +0000 UTC m=+2577.314862756" watchObservedRunningTime="2026-07-10 19:12:10.638294797 +0000 UTC m=+2577.316007258" Jul 10 19:12:12.832607 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:12.832574 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qlqq4_4fca207e-d050-4f78-beee-b9f61906bf18/dns/0.log" Jul 10 19:12:12.851808 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:12.851778 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qlqq4_4fca207e-d050-4f78-beee-b9f61906bf18/kube-rbac-proxy/0.log" Jul 10 19:12:12.972096 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:12.972071 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gnzkb_24f9d007-8b0e-4e0f-8062-3a66675027e8/dns-node-resolver/0.log" Jul 10 19:12:13.541422 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:13.541393 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b5dpv_b7b0a074-7ac6-4e87-9667-f1d0e4920459/node-ca/0.log" Jul 10 19:12:14.470102 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:14.470075 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6787xd5g_e2af4c51-c115-4f25-8868-4ab579f9633a/istio-proxy/0.log" Jul 10 19:12:14.597858 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:14.597827 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d88546674-cpbkv_7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1/kube-auth-proxy/0.log" Jul 10 19:12:14.653814 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:14.653781 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-npkgq_f3d5f957-c477-465a-8199-48833698b766/istio-proxy/0.log" Jul 10 19:12:14.710873 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:14.710793 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-npkgq_f3d5f957-c477-465a-8199-48833698b766/istio-proxy/1.log" Jul 10 19:12:15.555708 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:15.555678 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-75cm8_f5ff0e60-3cce-449d-ad20-f872820b48c2/serve-healthcheck-canary/0.log" Jul 10 19:12:16.297988 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:16.297961 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nrqp5_12d0fea4-abb3-467a-8422-7e1a81d723b3/kube-rbac-proxy/0.log" Jul 10 19:12:16.321639 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:16.321614 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nrqp5_12d0fea4-abb3-467a-8422-7e1a81d723b3/exporter/0.log" Jul 10 19:12:16.344854 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:16.344822 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nrqp5_12d0fea4-abb3-467a-8422-7e1a81d723b3/extractor/0.log" Jul 10 19:12:16.636401 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:16.636371 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w9pq4/perf-node-gather-daemonset-v8zjn" Jul 10 19:12:18.880774 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:18.880739 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-hfj7g_9600ebbe-f0bc-4265-884b-274efae32892/manager/0.log" Jul 10 19:12:19.113665 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:19.113573 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wkrqc_79685dda-acd4-4446-93b1-2608b69dd463/manager/0.log" Jul 10 19:12:19.284009 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:19.283977 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zjr4g_6dfdbad6-5aa2-4443-9398-3a59c736a589/server/0.log" Jul 10 19:12:19.392177 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:19.392101 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-czvzp_3be6a366-f512-4c82-887c-195e16cd0ea1/manager/0.log" Jul 10 19:12:19.610430 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:19.610392 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz_3bfd1a11-9e2b-4017-8996-699a64b4e0de/manager/0.log" Jul 10 19:12:19.627446 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:19.627422 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d4bb5bfc9-zsvcz_3bfd1a11-9e2b-4017-8996-699a64b4e0de/copy-manifests/0.log" Jul 10 19:12:20.577069 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:20.577042 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6787xd5g_e2af4c51-c115-4f25-8868-4ab579f9633a/istio-proxy/0.log" Jul 10 19:12:20.684993 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:20.684798 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d88546674-cpbkv_7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1/kube-auth-proxy/0.log" Jul 10 19:12:20.721798 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:20.721766 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-npkgq_f3d5f957-c477-465a-8199-48833698b766/istio-proxy/0.log" Jul 10 19:12:20.784715 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:20.784655 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-npkgq_f3d5f957-c477-465a-8199-48833698b766/istio-proxy/1.log" Jul 10 19:12:20.902060 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:20.901980 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-dsm6k_ae1ef643-3b34-449c-a8d5-4603d06458f3/openshift-lws-operator/0.log" Jul 10 19:12:22.106076 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:22.106042 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6787xd5g_e2af4c51-c115-4f25-8868-4ab579f9633a/istio-proxy/0.log" Jul 10 19:12:22.218805 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:22.218778 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d88546674-cpbkv_7c7e0f57-d7a2-428d-8dd1-3562d6ab25b1/kube-auth-proxy/0.log" Jul 10 19:12:22.257940 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:22.257910 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-npkgq_f3d5f957-c477-465a-8199-48833698b766/istio-proxy/0.log" Jul 10 19:12:22.315851 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:22.315823 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-npkgq_f3d5f957-c477-465a-8199-48833698b766/istio-proxy/1.log" Jul 10 19:12:27.138940 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:27.138912 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-94n7z_91835427-78a0-464d-87ab-d70b45a77388/kube-multus-additional-cni-plugins/0.log" Jul 10 19:12:27.158009 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:27.157983 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-94n7z_91835427-78a0-464d-87ab-d70b45a77388/egress-router-binary-copy/0.log" Jul 10 19:12:27.177250 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:27.177230 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-94n7z_91835427-78a0-464d-87ab-d70b45a77388/cni-plugins/0.log" Jul 10 19:12:27.196584 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:27.196563 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-94n7z_91835427-78a0-464d-87ab-d70b45a77388/bond-cni-plugin/0.log" Jul 10 19:12:27.217455 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:27.217427 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-94n7z_91835427-78a0-464d-87ab-d70b45a77388/routeoverride-cni/0.log" Jul 10 19:12:27.238113 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:27.238088 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-94n7z_91835427-78a0-464d-87ab-d70b45a77388/whereabouts-cni-bincopy/0.log" Jul 10 19:12:27.258340 ip-10-0-140-180 kubenswrapper[2576]: I0710 19:12:27.258316 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-94n7z_91835427-78a0-464d-87ab-d70b45a77388/whereabouts-cni/0.log"